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Preface 


This  monograph  grew  out  of  a  presentation  in  May  2013  at  a  special 
meeting  of  the  Military  Operations  Research  Society.  The  meeting, 
“The  Role  of  Analytics  in  Addressing  the  New  Budget  Environment,” 
discussed  current  challenges  facing  the  U.S.  Department  of  Defense 
and  how  analysis  could  assist  such  activities  as  the  Quadrennial  Defense 
Review.  This  longer  document  describes  my  view  of  how  analysis  can 
be  more  useful  than  it  sometimes  has  been  in  the  past  by  exploiting 
improvements  in  the  state  of  the  art  and  by  rediscovering  and  supple¬ 
menting  classic  principles.  It  reflects  my  experience  over  decades  as  an 
analyst  at  the  Institute  for  Defense  Analyses,  an  analyst  and  senior 
executive  in  the  Office  of  the  Secretary  of  Defense  with  responsibili¬ 
ties  for  both  strategy  and  program  analysis,  and — for  many  years — an 
analyst  and  manager  at  the  RAND  Corporation. 

The  monograph’s  primary  intended  audience  includes  defense 
analysts,  their  managers,  and  the  policymakers  who  are  consumers  of 
defense  analysis.  That  said,  most  of  the  principles  outlined  here  apply 
to  other  government  agencies  and  to  strategic  analysis  generally.  Com¬ 
ments  are  welcome  (email:  pdavis@rand.org). 

The  research  for  this  monograph  was  conducted  within  the  Inter¬ 
national  Security  and  Defense  Policy  (ISDP)  Center  of  the  RAND 
National  Defense  Research  Institute,  a  federally  funded  research 
and  development  center  sponsored  by  the  Office  of  the  Secretary  of 
Defense,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Navy, 
the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence 
Community.  For  more  information  on  the  RAND  ISDP  Center,  see 
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http://www.rand.org/nsrd/ndri/centers/isdp.html  or  contact  the  direc¬ 
tor  (contact  information  is  provided  on  the  web  page). 
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Summary 


Overview 

This  monograph  suggests  ways  for  higher-level  defense  analysis  to 
better  serve  the  needs  of  policymakers,  even  in  periods  of  austerity.  The 
suggestions  here  may  be  especially  significant  because  current  defense 
planning  also  has  many  strategic  challenges.  A  starting  point  is  to  see 
analysis  as  aiding  decisions,  as  suggested  in  Figure  S.l.  Starting  at  the 
bottom  of  the  figure,  one  sees  that  analysis  is  not  just  about  evaluating 
options  straightforwardly.  Rather,  it  must  (1)  ensure  that  a  good  set 
of  options  are  considered,  (2)  recognize  multiple  criteria  for  evaluat¬ 
ing  options,  (3)  confront  uncertainty  about  the  world,  and  (4)  expect 
disagreements  among  policymakers.  Despite  this  complexity,  analysis 

Figure  S.l 
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should  frame  and  compare  options  comprehensibly  with  a  premium  on 
simplicity  and  a  meaningful  “story.”  Simplifications,  however,  must  be 
approximately  valid.  Simplicity  is  also  a  relative  concept:  It  may  mean 
describing,  at  a  high  level,  how  options  deal  with  multiple  components 
of  a  system  problem  or  how  they  correspond  to  different  ways  to  bal¬ 
ance  a  portfolio  across  multiple  objectives,  including  risk-control  objec¬ 
tives.  Thus,  simple  should  not  be  simplistic. 

The  analysis  framework  should  recognize  that  decisions  often 
depend  on  considerations  beyond  what  analysis  provides.  Once  deci¬ 
sions  are  made,  analysis  should  help  policymakers  to  communicate, 
explain,  and  convince.  It  should  also  help  shape  implementation  guid¬ 
ance  with  sharpened  requirements,  forcing  functions,  and  metrics  for 
monitoring,  feedback,  and  adaptation. 

To  accomplish  these  aims  in  a  study,  it  is  wise  to  plan  an  analy¬ 
sis  campaign.  Experienced  analysis  managers  already  do  so,  but  what 
follows  is  an  enriched  conception  stemming  from  the  perspective  of 
capabilities-based  planning.* 

Capabilities-based  planning  is  planning  under  uncertainty  to 
provide  capabilities  for  a  wide  range  of  modern-day  challenges 
and  circumstances  while  working  within  an  economic  framework 
that  necessitates  choice. 

When  done  well,  then,  capabilities-based  planning  confronts  uncer¬ 
tainty  and  the  need  to  make  choices  within  constrained  budgets.  Prop¬ 
erly  understood,  it  has  always  considered  both  generic  possibilities  and 
specific  threats. 

Some  of  the  monograph’s  guidelines  on  analysis  campaigns  will 
be  familiar  and  even  old-hat  to  readers;  others  will  not  be.  The  intent  is 
to  suggest  best  practices  rather  than  introduce  radical  ideas,  although 
some  ideas  were  seen  as  radical  not  long  ago  and  others  may  still  be. 


What  I  describe  in  this  monograph  is  sometimes  referred  to  as  “capabilities-based  plan¬ 
ning  done  right”  because  implementation  of  the  concept  has  sometimes  been  troubled  (e.g., 
with  complex  bureaucratic  processes  and,  ironically,  too  little  emphasis  on  dealing  with 
uncertainty  and  making  choices).  Rather  than  invent  yet  a  new  term,  I  have  opted  simply  to 
define  my  usage.  See  also  Appendix  B. 
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Seeking  Flexibility,  Adaptiveness,  and  Robustness 

One  core  theme  in  an  analysis  campaign  should  be  confronting  “deep 
uncertainties”  such  as  those  spawning  what  some  call  “black  swan” 
events.  Another  is  dealing  with  multiple  objectives.  The  result  will 
inform  decisions  on  how  to  balance  and  hedge  when  planning.  Related 
to  this,  analysis  should  include  comparing  options  for  their  “FARness,” 
i.e.,  for  whether  they  provide  capabilities  allowing  for 

flexibility  to  take  on  new  or  changed  missions,  objectives,  .  .  . 

adaptiveness  to  new  or  changed  circumstances 

robustness  to  adverse  shocks  (or  even  highly  positive  shocks). 

This  sentiment  goes  by  such  varied  names  as  robust  decisionmaking, 
planning  for  adaptiveness,  and  planning  for  agility. 

Regardless  of  the  sticker  name,  this  approach  implies  a  new  pro¬ 
fessional  responsibility  for  analysts-.  Instead  of  merely  listing  analysis 
assumptions,  analysts  should 

•  routinely  show  how  results  vary  with  all  key  assumptions  and  dis¬ 
agreements — the  opposite  of  focusing  on  a  standardized  case  and 
perhaps  running  a  few  excursions 

•  routinely  assess  options  for  FARness,  showing  the  value  of  afford¬ 
able  hedges  even  in  periods  of  austerity  when  hedges  may  seem 
like  luxuries 

•  do  the  above  comprehensibly  to  aid  policymakers  in  converging  on 
decisions  and  actions. 

The  last  point  is  crucial  because  policymakers  need  good  summaries 
and  will  not  tolerate  hand-wringing  about  uncertainty  or  “paralysis  by 
analysis.” 


Simplicity  Versus  Depth 

An  analysis  campaign  will  often  need  a  mix  of  relatively  simple  and 
more  complex  models.  Suitable  low-resolution  models  are  particularly 
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good  for  “capabilities  analysis,”  i.e.,  exploratory  work  varying  param¬ 
eters  of  the  problem  simultaneously  to  generate  insights  and  tradeoffs. 
Such  models  also  frame  problems  with  the  higher-level  variables  suit¬ 
able  for  discussion  with  policymakers.  That  is,  they  provide  a  story. 
Higher-resolution  models  are  necessary  to  understand  issues  thor¬ 
oughly,  to  connect  with  real-world  data  and  operational  activities,  and 
to  reflect  subtleties.  Details  often  matter,  and  simplicity  is,  in  a  sense, 
only  a  necessary  fiction  along  the  way. 


Breadth 

The  analysis  campaign  should  provide  for  breadth  with  a  mix  of  models, 
human  gaming,  historical  analysis,  trend  analysis,  and  collaboration 
with  experienced  operators.  It  should  reflect  both  technology-push 
and  demand-pull.  Such  breadth  can  be  seen  as  including — beyond 
“normal”  analysis — lines  of  activity  with  features  akin  to  work  by  the 
Office  of  Net  Assessment  in  the  Office  of  the  Secretary  of  Defense 
(OSD),  combatant-commander  contingency  planners,  forward-looking 
planners  seeking  to  exploit  technology,  and  lessons-learned  studies. 


Multiobjective  Assessments  (Including  Risk  Management) 

An  analysis  campaign  should  identify  early  the  many  dimensions 
that  need  to  be  considered  in  constructing  and  evaluating  options. 
These  correspond  to  multiple  study-dependent  objectives  (including 
risk-management  objectives)  in  approximate  hierarchies  of  detail.  A 
strategic-level  study  might  have  separate  objectives  for  each  regional 
and  functional  area,  as  well  as  such  cross-cutting  challenges  as  simul¬ 
taneous  conflicts.  It  might  also  have  different  objectives  for  the  near, 
mid,  and  long  term.  A  mission-level  study  (e.g.,  of  the  capability  to 
improve  air  support  of  ground-force  operations  or  to  improve  cyber 
defenses)  might  have  different  objectives  for  each  mission-level  scenario 
with  objectives  reconsidering  military  effects,  collateral  damage,  and 
friendly  losses. 
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Exploratory  Capabilities  Analysis 

The  campaign  must,  of  course,  include  analysis  itself.  Although  analy¬ 
sis  organizations  commonly  focus  on  scenarios  established  by  policy 
offices,  an  important  analysis  role  is  to  help  identify  and  design  possible 
planning  scenarios,  discuss  implications  with  policymakers,  and  sub¬ 
sequently  tune  the  scenarios  so  that  they  accomplish  what  is  intended 
in  the  rest  of  the  planning  process.  Figure  S.2  describes  the  desired 
process  schematically,  a  process  with  good  historical  precedents  in 
Department  of  Defense  (DoD)  planning  over  the  years.  Blue  items  are 
analysis  tasks  that  should,  however,  be  accomplished  in  an  integrated 
partnership  (not  a  sequential  process)  among  OSD  (Policy),  OSD 


Figure  S.2 
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(Cost  Assessment  and  Program  Evaluation),  the  Joint  Staff,  and  OSD 
(Acquisition,  Technology,  and  Logistics).  These  offices  would  have  core 
responsibility,  but  many  aspects  of  the  process  should  be  as  open  and 
collaborative  as  possible,  something  much  valued  by  all  DoD  partici¬ 
pants.  The  services  would  have  major  roles  because  of  their  expertise, 
knowledge  of  relevant  data,  and  inherent  interests. 

Referring  to  numbered  items  in  the  figure,  in  this  approach,  ana¬ 
lysts  take  a  broad  scenario-space  view  to  recognize  the  many  objectives, 
constraints,  and  uncertainties  (Step  1).  This  stage  of  the  analysis  cam¬ 
paign  confronts  deep  uncertainties.  It  involves  “divergent  thinking” 
that  departs  from  standard  thinking  in  recognizing  issues  often  glossed 
over.  It  then  discovers  how  the  scenario  space  breaks  the  scenario  space 
into  regions  posing  different  challenges  (Step  2).  Depending  on  the 
circumstances  of  a  given  region,  the  challenges  might  involve,  e.g., 
response  time,  technical  capabilities,  force  size,  and  plausible  but  unex¬ 
pected  adversary  strategies.  The  challenges  might  also  be  political,  eco¬ 
nomic,  or  social,  as  in  “complex  endeavors”  in  general  or  as  illustrated 
in  recent  wars  in  particular.  The  next  part  of  the  analysis  campaign 
should  be  convergent.  Analysts  can  identify  representative  param¬ 
eterized  scenarios  for  each  challenge  region  (Step  3).  Given  suitable 
models,  they  can  then  do  first-cut  capabilities  analysis  to  estimate  what 
is  needed  to  meet  the  various  challenges  as  a  function  of  cost  (Step 
4).  Since  cost  and  feasibility  depend  on  the  stringency  of  challenges, 
parameterizing  stringency  (i.e.,  showing  implications  as  a  function  of 
stringency)  becomes  part  of  the  analysis. 

The  process  in  Figure  S.2  next  envisions  going  to  policymakers 
(top  yellow  diamond)  to  discuss  what  capabilities  they  wish  to  pursue 
further  given  results  of  the  first-cut  analysis.  Policymakers  then  make 
initial  decisions,  giving  up  on  some  capabilities  but  pursuing  others. 
That  is,  they  decide  tentatively  on  “requirements.”  This  requirement¬ 
setting  must  be  detailed  enough  to  define  intent.  Thus,  analysts  trans¬ 
late  qualitative  desires  (e.g.,  “deter,”  “achieve  an  early  halt,”  or  “achieve 
persistent  surveillance”)  into  parameter  settings  within  the  test-case 
scenarios  to  be  used  subsequently  (Step  5).  Parameter  settings  may 
differ,  for  example,  for  evaluation  of  Program  Objective  Memorandum 
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issues,  training  and  exercising,  and  suggestions  to  combatant  com¬ 
manders  for  operational  planning. 

In  response  to  guidance,  the  services  and  defense  agencies  develop 
various  proposals  and  evaluate  them  against  the  test  cases  and  uncer¬ 
tainty  ranges.  The  results  are  then  reviewed  and  analyzed  (Step  6)  to 
aid  decisionmakers  in  making  “final”  decisions  on  forces,  weapon  sys¬ 
tems,  and  other  matters.  Given  decisions,  analysts  then  adjust  prior 
guidance,  test-case  scenarios,  metrics,  and  goals  accordingly.  The  pro¬ 
cess  continues  and  further  iteration  occurs  in  subsequent  years  (not 
shown). 

Table  S.l  illustrates  capabilities  analysis  of  the  sort  assumed 
throughout  the  process  of  Figure  S.2.  It  uses  a  purely  notional  example 
of  comparing  two  options  for  homeland  ballistic-missile  defense  as  dis¬ 
cussed  in  Chapter  Three,  based  on  prior  publications.  The  message  for 
policymakers  is  that  Option  2  would  cost  twice  as  much,  but  its  value 
would  be  limited  largely  to  weak  threats  (the  bottom  two  rows).  Given 
a  tight  budget,  this  analysis  might  suggest  proceeding  with  Option  1 
while  continuing  research  and  development  on  the  capabilities  asso- 


Table  S.l 

Ballistic-Missile  Defense  Capabilities,  by  Threat  and  Objective  (Notional) 


Class  of  Threat  Option  1  ($100B)  Option  2  (S200B) 


Near-  Near- 

Minimum  Moderate  Perfect  Minimum  Moderate  Perfect 
Objective  Defense  Defense  Defense  Defense  Defense  Defense 


Massive  attack,  near 
peer 

Small  attack,  near  peer; 
or  multiple  missiles, 
advanced  rogue 

Multiple  missiles;  simple 
rogue 

Single  missile;  simple 
rogue 


SOURCE:  Davis,  Shaver,  and  Beck,  2008. 

NOTES:  Red  =  very  poor,  orange  =  poor;  yellow  =  medium,  light  green  =  good,  and  green  =  very 
good.  These  capabilities  might  be  quantified  in  terms  of  the  probability  of  intercepting  a  high 
fraction  of  attacking  missiles. 
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dated  with  Option  2  if  they  had  enough  upside  potential.  The  story 
explaining  such  a  decision  would  be  straightforward  from  the  table. 
Significantly,  this  analysis  preemptively  addresses  the  natural  “what-if?” 
questions  rather  than  focusing  simply  on  a  nominal  threat.  To  some, 
the  analysis  would  also  suggest  the  need  for  additional  (albeit  more 
expensive)  options  with  greater  potential  against  advanced  threats. 

In  earlier  years,  the  proposed  requirement  to  do  such  uncertainty- 
sensitive  parametric  analysis  was  resisted  by  those  in  analytic  organiza¬ 
tions,  who  argued  that  they  were  unable  to  develop  and  coordinate  the 
massive  databases  allegedly  needed.  Their  assertion,  however,  depended 
on  the  particular  models  used  and  the  processes  for  coordinating  data 
across  all  DoD  stakeholders.  The  ponderousness  of  the  models  and  pro¬ 
cess  have  not  paid  their  way  for  higher-level  decision-aiding,  although 
they  have  great  value  for  other  purposes  as  described  in  the  main 
text.  Organizations  should  modify  their  analytic  tools  and  processes 
to  permit  such  capabilities  analysis.  No  excuses  should  be  permitted, 
since  such  analysis  has  long  been  demonstrated  and  underlay  many  of 
DoD’s  major  capability  decisions  over  the  decades. 


Developing  Capability  Models 

The  “capability  models”  referred  to  above  (used  in  Steps  4  and  6  of 
Figure  S.2)  can  be  defined  as  follows: 

A  capability  model  is  a  causal  model  that  allows  us  to  understand 
how  the  ability  to  accomplish  a  mission  depends  on  system  vari¬ 
ables,  circumstances,  and  goals  expressed  parametrically. 

Such  a  model  contrasts  with,  say,  a  large  campaign  model  that  describes 
developments  over  time  in  a  single  scenario  with  a  single  set  of  assump¬ 
tions  represented  by  scores  of  parameters  and  complex  databases  that 
are  agreed  upon  but  highly  uncertain.  A  capability  model  can  be  devel¬ 
oped  from  scratch  by  someone  who  understands  the  problem  area, 
by  generating  “motivated  metamodels”  from  a  more  detailed  model 
(Chapter  Four),  or  by  using  a  “big  model”  (even  certain  campaign 
models)  with  modular,  multiresolution  features  that  allow  it  to  be  used 
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for  exploration.  All  three  approaches  are  well  precedented.  The  requi¬ 
site  skills  are  less  common  than  in  earlier  years,  but  there  is  no  shortage 
of  talent — even  if  some  “reeducation”  is  needed. 

A  special  challenge  occurs  when  the  model-building  involves 
social-science  considerations.  In  addition  to  applying  traditional  social- 
science  methods  (e.g.,  comparative  case  studies)  to  gain  insight,  several 
model-building  approaches  are  worth  mentioning,  as  discussed  with 
references  in  Chapter  Four.  First,  such  qualitative  models  as  factor 
trees  and  influence  diagrams  can  be  valuable  in  an  analysis  campaign. 
Second,  some  of  these  can  be  turned  into  capability  models  using 
recently  developed  methods.  Third,  campaign  models  can  include 
political  and  economic  considerations.  If  such  a  model  is  modular, 
transparent,  vetted,  and  possessed  of  some  multiresolution  features,  it 
would  be  a  good  candidate  for  certain  kinds  of  parametric  analysis  of 
social-science  issues. 


Developing  the  Options 

Evaluating  options  presupposes  having  options  to  evaluate.  An 
analysis  campaign  should  provide  a  suitable  range  of  options  as  part  of 
Step  4  in  Figure  S.2.  For  higher-level  analysis  (e.g.,  for  the  Secretary 
of  Defense),  this  will  include  creating  composite  options  from  lower- 
level  building  blocks.  The  composite  options  should  attend,  to  a  greater 
or  lesser  degree,  to  all  of  the  objectives  that  must  be  addressed.  The 
effort  to  provide  such  options  could  include  the  structured  use  of  inde¬ 
pendent  experts,  human  gaming,  and  requests  for  information  from 
industry.  Some  of  this  should  be  technology-push  in  nature  and  may 
call  for  major  changes.  Such  efforts  are  especially  important  in  periods 
of  austerity  when  doing  more  with  less  will  typically  require  a  combi¬ 
nation  of  using  technology  and  changing  both  concepts  of  operation 
and  organizations.  The  options  arising  in  more  usual  ways  may  be  less 
imaginative,  call  for  unacceptable  dropping  of  important  missions,  or 
have  organizations  clinging  to  as  much  of  their  legacy  structure  as  pos¬ 
sible,  rather  than  cutting  even  more  so  as  to  leave  room  for  innovation. 
This  problem  is  familiar  to  DoD  currently. 
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At  a  more  technical-analytic  level,  Chapter  Five  describes  a  new 
analytic  method  that  can  generate  a  vast  number  of  options  and  then 
filter  for  only  the  small  subset  that  are  potentially  attractive  in  the  mul¬ 
tiobjective,  uncertainty-laden  context.  The  method  generalizes  clas¬ 
sic  “efficient-frontier”  methods  to  deal  with  uncertainty  and  multiple 
objectives. 


Putting  It  All  Together:  Portfolio  Analysis  for  Integrative 
Decision  Support 

The  last  part  of  the  analysis  campaign  should  “put  things  together.” 
A  natural  mechanism  is  portfolio  analysis  of  options  that  vary  in  how 
they  use  mixes  of  instruments  to  address  multiple  objectives  while  work¬ 
ing  within  a  budget.  The  analogy  is  to  having  alternative  portfolios  of 
such  financial  instruments  as  stocks  and  bonds  to  deal  with  such  mul¬ 
tiple  objectives  as  long-term  capital  gain  and  reliable  current  income. 
For  DoD,  options  may  differ  in  the  relative  emphasis  on  ground,  air, 
or  naval  forces,  in  the  relative  emphasis  on  different  regional  and  func¬ 
tional  theaters  of  operation,  in  the  relative  emphasis  on  short-term  and 
longer-term  problems,  and  so  on.  Striking  the  right  balance  (which 
does  not  imply  evenness)  is  the  challenge  for  the  Secretary  of  Defense, 
President,  and  Congress. 

Analysis  should  therefore  discuss  strategic  options  so  that  pol¬ 
icymakers  can  see  how  the  options  deal  with  the  various  objectives 
(including  risk-control  objectives)  and  how  much  they  cost.  This  sug¬ 
gests  the  use  of  policy  scorecards  (even  the  sometimes-maligned  stop¬ 
light  charts)  rather  than  the  kinds  of  bar  charts  or  graphs  appropri¬ 
ate  in  other  types  of  analysis.  It  is  crucial,  however,  that  policymakers 
understand  why  the  options  perform  as  shown  in  the  scorecards.  They 
must  be  allowed  to  ask,  e.g.,  “Why  is  Option  2,  which  I  like,  per¬ 
forming  so  poorly  against  Objective  D?”  They  should  be  able  to  zoom 
in — even  within  a  briefing  context — to  see  the  underlying  logic.  The 
zoom  may  bring  up  another  scorecard  that  allows  visual  explanation 
at  a  glance.  For  example,  the  evaluation  may  depend  on  several  fac¬ 
tors,  most  of  which  are  favorable,  but  one  of  which  is  a  “killer,”  such  as 
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excessive  vulnerability  to  a  countermeasure  or  excessive  dependence  on 
a  notoriously  unreliable  ally.  If  the  policymaker  asks  about  changing 
an  assumption  (e.g.,  the  weight  given  to  a  worst-case  scenario),  analysts 
should  be  able  to  draw  on  their  previous  capabilities  analysis  to  show 
tradeoff  curves  or  other  responsive  depictions. 

Most  top  officials  will  use  the  zoom  option  only  rarely,  as  in  test¬ 
ing  the  mettle  of  staff  or  depth  of  analysis,  or  because  of  concerns  about 
a  particular  issue.  Deputies  and  staff,  however,  will  often  go  into  sub¬ 
stantial  detail.  Further,  experience  shows  that  structuring  the  analysis 
campaign  to  generate  material  for  such  a  layered  presentation  of  results 
is  an  excellent  way  to  ensure  solid  credible  analysis. 

Once  policymakers  have  made  choices,  analysis  can  help  to  tidy 
and  simplify.  As  one  example,  they  can  define  a  composite  measure  of 
effectiveness  and  generate  cost-effectiveness  charts  as  shown  schemati¬ 
cally  in  Figure  S.3.  This  represents  the  aggregate  significance  of  major 
uncertainties  and  disagreements  as  one  compares  investment  Options 
A,  B,  C,  and  D.  The  effectiveness  of  each  depends  on  underlying 
assumptions  that  are  assumed  to  cluster  in  two  “strategic  perspectives.” 
Perhaps  Perspective  I  reflects  an  emphasis  on  technology-push,  the 

Figure  S.3 

Cost-Effectiveness  Landscape,  by  Strategic  Perspective 
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future,  and  optimism  that  friendly  developments  will  outpace  counters. 
Perspective  II  might  be  more  near-term  oriented,  might  believe  that  the 
capabilities  of  Options  B,  C,  and  D  could  be  readily  countered,  and 
might  believe  that  Option  D  has  a  concept  of  operations  that  would  be 
counterproductive.  The  perspectives  agree  only  that  Option  A  is  well 
worth  the  investment.  As  a  variant,  even  someone  with  Perspective  I 
might  agree,  if  funds  were  tight,  to  stop  with  Option  B.  The  example 
is  notional,  but  dramatic  differences  across  perspective  have  been  dem¬ 
onstrated  in  past  studies  as  cited  in  the  main  text. 

Next,  we  come  to  implementation.  The  same  capabilities  modeling 
discussed  above  identifies  key  parameters  and  metrics  at  different  levels 
of  detail.  It  should  allow  analysts  to  fine-tune  the  nominal  parameter 
settings  and  ranges  in  the  test-case  scenarios  and  to  define  metrics  for 
follow-up  monitoring  and  adaptation.  This  is  no  small  matter,  since  it 
is  common  for  organizations  to  generate  metrics  in  more  ad  hoc  ways, 
which  often  creates  confusion  and  counterproductive  incentives. 

To  wrap  up  (see  also  Table  7.1  in  the  main  text),  the  monograph 
describes  an  approach  to  dealing  effectively  with  uncertainty.  The 
approach  envisions  demanding  more  from  higher-level  analysis  and 
analysts,  particularly  routine  evaluation  of  options  for  flexibility,  adap¬ 
tiveness,  and  robustness,  and  also  finding  simple  but  credible  ways  to 
aid  decisionmaking,  explaining  decisions  to  and  convincing  others, 
and  defining  implementation  plans  with  metrics.  All  this  will  require 
new  analytic  methods  with  reduced  dependence  on  detailed  models 
and  massive  databases,  although  those  should  remain  important  for 
establishing  the  common  base  of  knowledge  and  for  integrative  work. 
The  approach  urges  streamlined  processes  in  which  select  analysts  from 
OSD  (Policy),  OSD  (Cost  Assessment  and  Program  Evaluation),  the 
Joint  Staff,  and  OSD  (Acquisition,  Technology,  and  Logistics)  work 
together  rather  than  sequentially.  They  would,  of  course,  continue  to 
depend  heavily  on  the  services  and  combatant  commands  for  expertise 
and  suggestions. 
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CHAPTER  ONE 


A  Setting  of  Great  National  Security  Challenges 


Fiscal  Constraints  Are  Only  Part  of  the  Problem 

As  of  late  2013,  the  U.S.  Department  of  Defense  (DoD)  was  still  strug¬ 
gling  with  severe  budget  cuts.  Historically,  a  10  percent  cut  in  the  DoD 
program  has  led  to  a  20  percent  cut  in  force  structure  due  to  other 
expenses  being  difficult  to  understand,  isolate,  and  trim.1  Also,  certain 
costs  such  as  those  associated  with  long-term  medical  benefits  have 
risen  markedly,  and  the  inflation-corrected  cost  per  member  of  the 
military  continues  to  grow  as  it  has  for  decades.  These  increases  further 
complicate  efforts  to  achieve  desired  force  levels.  Such  problems  have 
been  exacerbated  by  sequestration  and  related  uncertainty,  although 
the  sequestration  problems  were  mitigated  by  congressional  decisions 
in  December  2013. 

These  facts  are  part  of  the  context,  but  the  United  States  has  pro¬ 
found  national  security  challenges  of  which  the  budget  is  only  one. 
Figure  1.1  identifies  challenges  as  described  in  an  earlier  publication 
with  colleague  Peter  Wilson.2  If  our  earlier  work  is  largely  correct,  then 
the  task  of  analysis  will  be  even  greater  than  otherwise  because  new 
options  and  difficult  choices  will  be  needed  and  their  evaluation  will  be 
difficult.  We  argued  that  the  United  States  faces  a  mix  of  complex  and 
traditional  military  challenges;  extreme  difficulties  in  force  projection; 
block  obsolescence  of  U.S.  military  strategy,  force  structure,  and  con¬ 
cepts  of  operations;  and  the  need  for  a  new  grand  strategy  in  the  Asia- 
Pacific  region.  Obstacles  to  dealing  with  these  challenges  include  cur¬ 
rent  wars  and  crises,  complacency  about  obsolescence,  and,  of  course, 
fiscal  constraints.  Another  obstacle  is  striking:  the  absence  of  a  coherent 
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Figure  1.1 

Problems  Go  Far  Beyond  Fiscal  Constraints 


The  Impending  Crisis  in 
Defense  Planning 


SOURCE:  Davis  and  Wilson,  2011a. 
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vision  on  the  way  ahead  (despite  many  good  ideas  and  initiatives).  In 
the  period  1996-2001,  forward-leaning  planners  had  concrete  propos¬ 
als  to  transform  U.S.  forces  by  exploiting  precision  fires,  precision  navi¬ 
gation,  networking,  and  stealth  technologies.3  Doing  so  would,  it  was 
argued,  allow  performing  many  missions  better  than  before  but  with 
smaller  military  forces  and  corresponding  reductions  in  cost.  Many  of 
these  changes  have  since  occurred  and  are  now  taken  for  granted.  But 
no  analogous  and  coherent  proposals  loom  large  in  the  current  defense 
debates,  although  many  ingredients  are  present.  It  will  be  an  interest¬ 
ing  period  for  defense  analysis. 

Changes  in  U.S.  capability  to  project  forces  are  particularly  trou¬ 
blesome.  Drawing  again  from  the  earlier  publication,  Table  1.1  elabo¬ 
rates  for  eight  operations.  Some  are  widely  discussed  under  the  rubric 
of  anti- access  and  area  denial;  others  are  less  appreciated.  For  example, 
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Table  1.1 

U.S.  Capabilities,  Then  and  Now 


Operation 

Then:  Assessment 
in  Near  Past 

Now:  Assessment 
Looking  Forward 

Forward  presence 

7 

5 

Large  forward  deployments 

7 

3 

Broad  naval  supremacy 

7 

5 

Air  supremacy 

7 

5 

Suppression  of  enemy  air  defenses 

5 

3 

Offensive  air  operations 

7 

3 

Ground-force  entry 

7 

Later  ground-force  operations 

7 

3 

NOTE:  Red  =  very  poor,  orange  =  poor,  yellow  =  medium,  light  green  =  good,  and 
dark  green  =  very  good. 


the  United  States  may  not  be  able  to  count  on  easy  air  supremacy  due 
to  mobile  and  man-portable  surface-to-air  missiles.  Also,  ground-force 
operations  with  traditional  U.S.  units  will  be  more  dangerous  as  even 
“rogue-level”  adversaries  gain  some  precision-guided  weapons  and  area 
weapons.  Interestingly,  this  shift  from  enjoying  military  overmatch  to 
being  faced  with  numerous  difficult  challenges  has  long  been  predicted. 
A  1998  paper  looked  forward  to  what  it  called  Era  A  in  which  the  U.S. 
military  would  benefit  greatly  from  technological  developments,  but 
it  also  foresaw  a  very  difficult  Era  B  ahead  starting  in  about  2012.  By 
then,  China  would  have  arisen  rather  than  be  rising,  some  rogue  states 
would  have  nuclear  weapons  rather  than  be  working  on  them,  and  the 
fruits  of  technological  developments  would  be  more  broadly  dissemi¬ 
nated.4  We  are  now  in  Era  B.5. 


A  Pollyannish  Perspective  on  Tight  Budgets? 

Ironically,  this  background  of  challenges  may  suggest  that  budget 
stringency  could  be  helpful  since  the  need  for  tough  budget  choices  has 
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sometimes  forced  modernizations  and  organizational  changes  that  oth¬ 
erwise  would  have  been  successfully  resisted.  Better  that  change  come 
about  by  budget  necessity  than,  say,  by  losing  a  war.  Forward-looking 
choices  are  much  easier,  however,  when  good  options  are  on  the  table 
and  when  an  organization  has  latitude  in  how  to  spend  its  resources. 
These  conditions  do  not  apply  today,  with  the  result  that  many  current 
discussions  are  primarily  about  how  to  take  large  cuts  rather  than  how 
to  solve  the  problems  in  Table  1.1. 6  Nonetheless,  I  proceed  as  though 
the  situation  will  improve  and  that  analysis  can  help  DoD  make  far¬ 
sighted  choices.  The  question  is  how  analysis  can  do  so.  The  answer  is 
that  many  elements  of  and  process  for  analysis  need  to  be  rethought. 
What  follows  describes  my  own  suggestions. 
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CHAPTER  TWO 


Analysis  to  Aid  Decisionmaking 


A  starting  point  for  discussion  is  to  ask  what  policymakers  should 
expect  from  analysis. 


Classic  Tenets  Remain  Valid 

Defense  analysts  have  benefited  for  decades  from  tenets  expressed 
by  Alain  Enthoven,  who  founded  the  Office  of  Systems  Analysis  for 
Secretary  of  Defense  Robert  McNamara.  The  office  evolved  into  the 
Office  of  Program  Analysis  and  Evaluation  and,  most  recently,  Cost 
Assessment  and  Program  Evaluation.  Although  systems  analysis  was 
extremely  controversial  in  the  early  years,1  and  the  Vietnam  War  tar¬ 
nished  the  reputation  of  Secretary  McNamara,  a  remarkable  consensus 
developed  about  Enthoven’s  analysis  tenets  as  listed  below.2 3 4 5  Indeed, 
chiefs  of  the  military  services,  heads  of  other  government  agencies,  and 
corporate  managers  often  embrace  variants  tailored  to  their  purposes. 

1.  Decisions  should  be  based  on  explicit  criteria  of  national  inter¬ 
est,  not  on  compromises  among  institutional  forces. 

2.  Needs  and  costs  should  be  considered  simultaneously. 

3.  Major  decisions  should  be  made  by  choices  among  explicit,  bal¬ 
anced,  feasible  alternatives. 

4.  The  Secretary  of  Defense  should  have  an  active  analytic  staff  to 
provide  him  with  relevant  data  and  unbiased  perspectives. 

5.  A  multiyear  force  and  financial  plan  should  project  the  conse¬ 
quences  of  present  decisions  into  the  future. 
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6.  Open  and  explicit  analysis  (including  transparent  data  and 
assumptions),  available  to  all  parties,  should  form  the  basis  for 
major  decisions. 


Additional  Tenets  Are  Needed 

The  classic  tenets  have  held  up  well,  but  I  believe  that  additional  tenets 
are  needed. 

7.  Decisions  should  confront  “deep”  uncertainty  and  disagreement. 

8.  The  planning  process  should  ensure  creative  and  effective  col¬ 
laboration  across  strategy,  operations,  technology,  programs, 
and  budgets. 

9.  The  planning  process  should  provide  for  monitoring,  feedback, 
and  adaptation. 

I  will  discuss  each  of  these  in  turn  and  then  focus  on  the  first 
(Tenet  7). 

Deep  Uncertainty  and  Disagreement.  Tenet  7  is  important  because 
organizations  so  often  settle  into  routines  that  focus  on  alleged  best 
estimates,  official  forecasts,  official  scenarios,  and  prevailing  assump¬ 
tions.  The  history  of  strategic  planning,  however,  reveals  the  signifi¬ 
cance  of  uncertainties,  many  of  which  are  “deep,”  referring,  for  exam¬ 
ple,  to  how  the  future  will  unfold,  how  humans  will  behave,  and  other 
matters  for  which  there  are  no  reliable  probability  distributions.3  Actual 
developments  are  sometimes  very  different  from  those  expected  due 
to  unplanned-for  events  or  a  misunderstanding  of  the  problem.  Such 
uncertainties  are  all  too  often  poorly  treated  if  addressed  at  all.4  This 
is  now  recognized  even  in  books  about  “black  swans”  intended  for  air¬ 
plane  reading  by  managers.5  It  has  also  been  the  reason  that  U.S.  poli¬ 
cymakers,  including  defense  secretaries,  are  sometimes  disappointed 
in  the  analysis  they  receive.  This  was  reflected  in  2006  with  leader¬ 
ship  complaints  about  how  the  analytic  community  was  continuing 
to  focus  on  old  and  familiar  scenarios  rather  than  assisting  leadership 
with  irregular  warfare  and  exploring  the  effect  of  potential  strategic 
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shocks.6  To  be  sure,  exceptional  efforts  are  sometimes  made  to  com¬ 
pensate,  as  when  OSD  (Policy)  conducted  some  “trends-and-shocks” 
studies  between  2005  and  2008. 7  Such  efforts,  however,  are  outside  the 
mainstream  of  analysis. 

Deep  uncertainty  underlies  many  current  questions:  Will  China 
become  increasingly  aggressive  and  hegemonic?  Will  North  Korea  or 
Iran  actually  use  nuclear  weapons  at  some  point  despite  that  seeming 
to  us  irrational?  Will  violent  Islamist  Jihad  fade  away,  grow,  or  remain 
a  chronic  threat?  Will  future  special-operations-forces-centric  counter¬ 
insurgency  operations  be  successful  or  counterproductive?  Will  U.S. 
efforts  to  maintain  a  power  balance  in  the  Asia-Pacific  region  (e.g.,  the 
air-sea  battle)  be  effective  or  counterproductive?  The  list  of  such  unan¬ 
swerable  questions  is  long,  but  much  can  be  done  to  inform  decision¬ 
making  amid  such  uncertainty. 

Collaboration.  One  lesson  from  experience  is  that  problem-solving 
often  requires  dynamic,  stressful  collaboration  among  those  associated 
with  strategy  and  policy,  operations,  technological  developments,  and 
resource-sensitive  planning.  Stovepiped  organizations  often  mandate 
sequentially  cooperative  processes,  but  solutions  can  emerge  faster  and 
better  if  people  work  together  simultaneously  across  stovepipes.  To  use 
language  from  command  and  control  research,  success  is  often  enabled 
by  collaboration  and  networking  across  hierarchical  columns,  rather 
than  by  occasional  “deconfliction.”8  Unfortunately,  some  of  DoD’s 
processes  are  stovepiped  and  sequential  for  historical  and  organiza¬ 
tional  reasons,  as  well  as  interpretations  of  law.9  Interestingly,  many  of 
the  most  important  DoD  program  developments  have  occurred  only 
by  working  around  these  processes  (e.g.,  with  highly  classified  “black” 
programs).  This  said,  communication  and  collaboration  do  occur  rou¬ 
tinely  on  many  matters,  whether  because  of  personal  relationships 
among  senior  leaders,  inclusive  processes  (at  the  expense  of  having  large 
committees),  or  necessity. 

Monitoring,  Feedback,  and  Adaptation.  The  implementation  of 
plans  sometimes  proceeds  as  though  the  plans  are  reasonably  solid  and 
enduring.  Realistically,  of  course,  initial  plans  are  often  flawed,  poorly 
executed,  or  overtaken  by  events.10  Thus,  feedback  loops  are  essen¬ 
tial.  Their  importance  has  been  particularly  evident  to  DoD  in  the 
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last  decade  as  the  result  of  continual  wars  with  many  unanticipated 
developments.11  Fortunately,  the  U.S.  military  services  are  learning 
organizations  that  evolve  and  adapt — especially  when  circumstances 
necessitate  doing  so.  Nonetheless,  neither  DoD  nor  other  government 
agencies  are  good  at  routinely  building  into  programs  the  means  by 
which  to  monitor  outcomes,  reassess,  and  adapt.  This  is  a  work  con¬ 
stantly  in  progress.12 

Tenets  9  and  10  are  large  subjects  better  treated  elsewhere,  but 
Tenet  7  relating  to  planning  is  at  the  core  of  this  monograph.  It  leads 
to  a  new  principle  and  implications  for  analysts. 


A  New  Principle  and  Implications  for  the  Analysis 
Profession 

The  FARness  Principle 

The  last  two  decades  have  spawned  methods  for  dealing  effectively 
with  both  normal  and  deep  uncertainty.  They  suggest  a  new  principle, 
which  is  to  plan  (whether  for  future  force  structure  or  operations)  for 
flexibility,  adaptiveness,  and  robustness.  This  is  what  I  have  called  the 
FARness  principle.13  It  has  an  immediate  implication  for  analysis. 

•  Analysis  should  help  leaders  find  strategies  that  are  flexible,  adap¬ 
tive,  and  robust: 

-  flexible  to  accommodate  changes  of  mission,  objectives,  and 
constraints 

-  adaptive  to  circumstances 

-  robust  to  events  such  as  positive  or  negative  shocks 

•  Leaders  should  demand  analysis  that  does  so. 

This  basic  concept  could  have  been  embraced  by  any  Secretary 
of  Defense  in  memory.14  It  has  broad  support  among  those  who  plan 
under  uncertainty,  although  terminology  varies.  The  principle  overlaps 
with  what  has  elsewhere  been  called  planning  for  adaptiveness,  robust 
adaptive  planning,  robust  decisionmaking,  and  planning  for  agility. 
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A  considerable  literature  documents  applications  of  one  or  another 
variant.13 

Why  FARness  Is  Neither  Common  Sense  Nor  Easily  Obtained 

Elevating  FARness  to  a  principle  might  seem  merely  to  glorify  common 
sense:  Who  would  not  want  to  plan  for  FARness?  The  following  exam¬ 
ples  may  illustrate  why  the  matter  is  nontrivial. 

1.  Acquisition.  The  norm  in  acquisition  has  been  for  industry  to 
receive  precise  requirements  (although  requirement  ranges  are 
becoming  more  common),  even  though  the  future  uses  of  what 
is  being  acquired  and  the  capabilities  that  will  be  needed  may  be 
impossible  to  forecast  accurately.  Why  does  DoD  not  ask  indus¬ 
try  to  write  proposals  that  compete  in  the  versatility  of  their 
offerings  by  providing  parametric  descriptions  of  what  capabili¬ 
ties  can  be  delivered  as  a  function  of  cost  and  yet-to-be-decided 
requirements?16  The  government  would  then  better  understand 
the  tradeoffs,  and  industry  would  not  need,  when  writing  pro¬ 
posals,  to  “divine”  the  product  and  price  tag  that  will  eventually 
be  acceptable.  This  approach,  however,  is  unusual.  As  an  exam¬ 
ple  of  non-FARness,  consider  the  failure  over  decades  to  protect 
satellites,  aircraft,  and  other  systems  with  modern  electronics 
against  electromagnetic  pulse  effects,  which  created  major  vul¬ 
nerabilities  that  could  have  been  avoided  affordably.17  A  second 
well-known  example  is  that  initial  development  of  the  F-15  did 
not  allow  for  air-to-ground  capability — capability  that  was  later 
recognized  as  necessary.  Similar  disputes  have  arisen  in  the  his¬ 
tory  of  the  F-22  program. 

2.  Tyranny  of  Priority  Lists.  Establishing  priorities  is  essential,  but 
the  common  management  tactic  of  establishing  rank-ordered 
priority  lists  can  lead  to  bad  decisions  inconsistent  with  FAR¬ 
ness.  In  an  austere  budget  environment,  the  result  may  be  to 
fund  items  from  the  top  down  until  the  budget  is  exhausted. 
Since  the  most  expensive  items  sometimes  have  the  highest  pri¬ 
orities,  fully  funding  them  may  eliminate  items  for  capability 
that  will  prove  crucial.  Yes,  priority-setting  is  essential,  but  it 
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is  best  approached  in  the  language  of  “balancing  portfolios”  as 
discussed  in  Chapter  Six. 

3.  Difficulties  in  Attaining  Jointness.  DoD  has  been  remarkably  suc¬ 
cessful  in  promoting  jointness  over  the  last  25  years,  but  doing  so 
has  entailed  substantial  and  sustained  leadership  effort  and  the 
goals  are  not  yet  fully  implemented.18  For  example,  the  military 
components  continue  to  have  command  and  control  systems 
with  interoperability  problems  that  hinder  flexibility,  adaptive¬ 
ness,  and  robustness  in  operations.  Technical  difficulties  and 
specialized  needs  are  an  important  root  cause,  but  so  also  is  the 
fact  that  the  individual  military  services — when  squeezed  for 
budget  and  faster  acquisition — conclude  that  it  is  more  impor¬ 
tant  to  acquire  their  own  systems  than  to  ensure  interoperabil¬ 
ity.  Such  decisions  represent  locally  rational  decisions  “necessi¬ 
tated”  by  shortcomings  in  the  complex  overall  planning  system. 

Despite  these  problems,  DoD  has  generally  avoided  the  most  seri¬ 
ous  blunders  that  might  have  been  made.  Moreover,  it  has  sometimes 
demonstrated  flexibility,  adaptivity,  and  agility.  For  example,  after  the 
attacks  of  September  11,  2001,  it  was  able  to  employ  both  Army  and 
Marine  forces  to  landlocked  Afghanistan  without  previously  developed 
doctrine.  Special  operations  forces  on  the  ground  innovated  to  find 
ways  to  direct  precision  strikes  from  the  air.  Ground  forces  were  sup¬ 
ported  with  aircraft  rather  than  traditional  heavy  artillery. 

DoD’s  successes  in  this  regard  have  been  due  significantly  to  the 
quality  of  its  people  and,  on  the  planning  front,  its  large  budget,  decen¬ 
tralized  decisionmaking  that  has  allowed  advocates  of  new  ideas  to 
shop  around  for  buyers,  and  a  U.S.  tradition  of  supporting  research 
such  as  that  conducted  by  the  Defense  Advanced  Research  Projects 
Agency.19  Unfortunately,  current  budget-tightening  and  sequester- 
related  constraints  are  endangering  all  of  these  and  the  long-range  con¬ 
sequences  could  be  severe. 

Despite  this  worrisome  background,  in  what  follows  I  assume 
that  budgets  and  budget  processes  will  allow  reasonable  decisions  and 
return  to  implications  for  analysis  and  analysts. 
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New  Professional  Responsibilities  for  Analysts 

The  FARness  principle  implies  a  new  concept  of  the  analyst’s  profes¬ 
sional  responsibilities.  To  set  the  stage,  we  should  recall  that  one  of  the 
most  important  accepted  responsibilities  of  an  analyst  is  to  make  the 
assumptions  of  analysis  known  to  the  leaders  to  whom  the  analysis  is 
reported.  The  need  to  do  so  is  taught  routinely  to  young  analysts  and 
stressed  by  study  leaders.  It  is  an  excellent  principle — but  nowhere  near 
sufficient.  The  FARness  principle  suggests  the  following  additional 
professional  obligations: 

•  routinely  identify  and  assess  options  for  FARness,  showing  the 
value  of  affordable  hedges  to  policymakers  even  in  periods  of  aus¬ 
terity  when  hedges  may  seem  like  luxuries 

•  routinely  show  how  results  vary  with  the  major  assumptions  on 
which  there  is  or  should  be  disagreement;  it  is  not  sufficient  to 
show  sensitivities  to  only  one  or  a  few  issues  while  ignoring  others 
that  are  also  important.20 

A  subtlety  here  is  the  difference  between  analysts  who  fight  the 
policymaker’s  strategy  and  analysts  who  support  the  strategy  but  also 
identify  hedges.  Even  strong-willed  policymakers  can  appreciate  the 
latter,  so  long  as  the  hedging  does  not  endanger  the  strategy.  Many 
examples  of  possible  hedges  come  to  mind  drawn  from  historical  expe¬ 
rience.  These  include  (1)  continuing  research  and  development  on  a 
weapon  system  that  has  been  ruled  out  for  the  present  but  has  upside 
potential  for  the  future,  (2)  preserving  the  ability  to  reopen  an  assem¬ 
bly  line,  (3)  maintaining  cadre-staffed  reserve-component  units  rather 
than  disbanding  them,  (4)  mothballing  ships  rather  than  destroying 
them,  (5)  acquiring  more  drones  for  intelligence,  surveillance,  and 
reconnaissance  than  are  currently  thought  to  be  needed,  (6)  maintain¬ 
ing  infrastructure  for  recruiting,  training,  and  equipping  additional 
ground  forces  if  in  the  future  the  Army  and  Navy  need  to  be  scaled 
up  again,  (7)  accepting  deliberate  redundancy  in  programs  or  forces  to 
reduce  risks  (e.g.,  by  being  able  to  employ  either  ground-based  or  sea- 
based  air  forces  as  a  crisis  arises)  and  foster  competition,  and  (8)  using 
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an  economy-of-force  strategy  in  a  region  rather  than  pulling  forces  out 
of  the  region  and  leaving  a  possible  vacuum. 

Hedges  are  not  free,  sometimes  undercut  strategy,  and  are  some¬ 
times  counterproductive  by  delaying  more  decisive  actions.  Some 
hedges  are  wise;  some  are  not.  Analysts  can  help  find  the  good  ones 
and  some  analysts  do  so  routinely.  Often,  however,  doing  so  is  not  seen 
as  an  important  function  of  analysis,  especially  if  “requirements”  are 
imagined  to  be  fully  adequate  and  solid. 

With  this  background  in  objective-setting  for  analysis,  the 
remainder  of  the  monograph  discusses  what  analysis  should  arguably 
look  like  and  ideas  for  making  that  vision  into  a  reality. 
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which  may  seem  logical  but  is  troublesome  (see,  e.g.,  Chapter  Two  in  Davis,  Shaver, 
and  Beck,  2008,  with  excerpts  appearing  also  in  Daviset  al.,  2008. 
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10  Mintzberg  (1994).  Parts  of  the  business  world  have  moved  on  with  attitudes  con¬ 
sistent  with  those  in  this  monograph  (Grant,  2002,  2010). 

11  Examples  have  included  the  insurgency  in  Iraq,  the  effect  of  improvised  explosive 
devices  leading  to  rapid  acquisition  of  the  mine-resistant,  ambush-protected  vehicle 
(Rogers,  2012);  the  ineffectiveness  of  early  U.S.  counterinsurgency  strategy  and  the 
overhaul  of  doctrine  by  General  David  Petraeus  (Army)  and  General  James  Mattis 
(Marine  Corps)  (Petraeus  and  Amos,  2006);  huge  cost  increases  for  weapon  systems 
such  as  the  F-22  and  F-35;  and  technologically  inspired  innovations  in  direct- action 
operations  with  modern  networking  (McChrystal,  2011). 

12  This  comment  reflects  ongoing  work  by  colleague  Jeff  Drezner  for  DoD’s  Office 
of  the  Under  Secretary  for  Acquisition  and  Technology,  as  well  as  such  examples 
in  homeland  security  work  noting  the  need  for  programs  to  monitor  and  evaluate 
effectiveness  (Jackson  et  ah,  2012)  and  to  have  peer-reviewed  assessments  of  the 
results  (Davis  et  ah,  2013).  The  U.S.  Government  Accountability  Office  has  pub¬ 
lished  a  series  of  reports  that  describe  its  perspective  on  related  DoD  acquisition 
efforts  to  plan  for  important  “knowledge  points”  at  which  decisions  can  be  made 

(2013). 

13  The  terminology  dates  back  to  1993-1994  with  the  terms  having  distinct  clas¬ 
sic  meanings  (Davis  and  Finch,  1993;  Davis,  1994a,  1994b).  In  everyday  speech, 
however,  the  terms  are  often  synonyms.  Some  of  the  themes  were  repeated  in  a  later 
collection  (Johnson,  Tibicki,  and  Treverton,  2003). 

14  Robert  McNamara  commissioned  a  1963  study  that  looked  at  a  wide  range  of 
possible  conflict  scenarios.  Only  afterward  was  it  concluded  that — for  force-sizing 
purposes — simplification  was  possible  in  terms  of  worse-than-expected  cases.  Sub¬ 
sequent  Secretaries  have  routinely  requested  analysis  across  scenarios,  but  organiza¬ 
tions  have  resisted  going  beyond  the  canonical  scenarios  of  the  time. 

15  A  considerable  policy-analysis  literature  exists  from  RAND  (Davis,  1994b,  2012; 
Lempert,  Schlesinger,  and  Bankes,  1996;  Lempert  et  al.,  2006)  and  Delft  Univer¬ 
sity  in  the  Netherlands  (van  de  Riet,  2003).  See  also  defense-related  discussion  from 
the  Hague  Centre  for  Strategic  Studies  (De  Spiegeleire,  2011)  and  NATO’s  com¬ 
mand  and  control  research  (Alberts,  2011).  See  also  a  RAND  website  collecting 
publications  using  robust  decisionmaking  methods,  mostly  in  social-policy  contexts 
(http://www.rand.org/topics/robust-decision-making.html). 

16  This  discussion  benefited  from  discussions  with  colleague  Elliot  Axelband. 

17  Foster  et  al.,  2008. 

18  Harold  Brown  has  recounted  early  history,  noting  roots  to  President  Eisenhower 
(Brown  and  Winslow,  2012,  p.  54  ff).  Congress  had  a  powerful  role  with  passage 
of  the  Goldwater-Nichols  legislation  in  1986.  General  Colin  Powell  took  an  activist 
stance  implementing  the  intent  during  his  tenure  as  Chairman  (Jaffe,  1993;  Powell 
and  Persico,  1995).  Donald  Rumsfeld’s  shift  to  capabilities-based  planning  was  a 
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major  step  in  institutionalizing  change  (Joint  Defense  Capabilities  Study  Team, 
2004). 

19  One  notable  example  was  JSTARS,  the  Joint  Surveillance,  Tracking,  and  Recon¬ 
naissance  System  platform,  which  might  never  have  been  acquired  except  that  a 
Defense  Advanced  Research  Projects  Agency  prototype  was  brought  into  opera¬ 
tional  use  in  the  1991  Gulf  War,  where  it  proved  the  concept’s  value  (Dunn,  Bing¬ 
ham,  and  Fowler,  2004). 

20  See  also  Dewar,  2003,  including  a  discussion  of  hedges.  More  commonly,  the 
emphasis  is  merely  on  listing  the  assumptions  and  perhaps  doing  some  excur¬ 
sions.  As  merely  one  example,  NATO’s  worthy  code  of  best  practice  for  command 
and  control  analysis  mentions  varying  assumptions  (p.  43),  but  more  prominent 
portions  of  the  text  convey  a  sense  more  of  the  need  merely  to  specify  them  (e.g., 
pp.  7,  64,  75,  101).  Similarly,  where  the  study  admonishes  varying  assumptions 
(p.  115),  the  language  used  is  with  the  word  “may,”  as  in  “Sensitivity  analysis  may 
be  applied.  .  .  .” 


CHAPTER  THREE 


Concrete  Suggestions  for  Analysis 


This  and  subsequent  chapters  address  specific  concepts  and  issues  that 
arise  in  attempting  to  plan  under  uncertainty.  The  first  is  the  concept 
of  conceiving  of  an  analysis  campaign  with  many  elements.  This  raises 
a  number  of  issues  that  are  discussed  in  terms  of  how  to  (1)  think 
about  analysis  for  decision-aiding,  (2)  balance  the  simple  and  complex, 
(3)  broaden  the  character  of  analysis,  (4)  deal  with  uncertainties  and 
disagreements,  (5)  recognize  multiple  objectives  and  criteria,  including 
the  need  to  manage  risk,  and  (6)  show  analytic  results  as  a  function  of 
major  assumptions  (exploratory  analysis). 

Such  analysis  often  requires  models  and  Chapter  Four  addresses 
how  to  develop  “capability  models”  enabling  the  style  of  analysis  called 
for  in  this  monograph.  Chapter  Five  then  turns  to  the  challenge  of 
finding  the  appropriate  options  to  assess.  Chapter  Six  then  describes 
portfolio-analysis  techniques  for  comparing  options.  It  includes  a  gen¬ 
eralized,  uncertainty-sensitive  version  of  cost-effectiveness  analysis  that 
is  especially  salient  for  a  period  of  austerity. 


The  Concept  of  an  Analysis  Campaign 

Figure  3.1  suggests  the  concept  of  an  analysis  campaign  schematically. 
Such  a  campaign  might  be  tailored  to  a  particular  upcoming  decision 
or  to  continuing  analysis  for  a  capability  or  mission  area  of  continued 
interest  and  development  (e.g.,  undersea  warfare,  defense  suppression, 
ground-force  maneuver).  The  campaign  might  have  extensive  original 


15 


16  Analysis  to  Inform  Defense  Planning  Despite  Austerity 


Figure  3.1 

Decision-Aiding  That  Draws  on  an  Analysis  Campaign 
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work  or  might  consist  largely  of  distilling  knowledge  available  from  the 
services,  industry,  academia,  and  elsewhere.1 

When  the  time  comes  for  analysts  to  discuss  results  with  poli¬ 
cymakers,  what  goes  forth  should  seldom  be  “the  answer”;  it  should 
instead  be  a  combination  of  a  well-framed  and  coherent  story  and  the 
ability  to  zoom  into  detail  to  explain  results  and  highlight  uncertain¬ 
ties  and  to  answer  diverse  and  perhaps  unstructured  questions  requir¬ 
ing  in-depth  knowledge.  While  most  responses  should  be  possible 
immediately  by  virtue  of  preparation,  a  fast  reach-back  capability  will 
sometimes  be  necessary. 

Some  of  what  is  depicted  in  Figure  3.1  occurs  routinely  in  analysis 
organizations.  Other  aspects  do  not.  In  particular,  there  has  often  been 
insufficient  emphasis  on  the  simple  story,  the  optional  zooms  for  expla¬ 
nation,  uncertainty  analysis,  and  related  parametric  tradeoffs.  Also, 
analysis  campaigns  are  sometimes  overly  tilted  toward  the  kinds  of 
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knowledge  obtained  from  big  computer  models.  And,  as  discussed  in 
the  next  section,  analysis  campaigns  are  often  construed  too  narrowly. 


Supporting  Policymakers:  It  Is  Not  Just  Decision-Aiding 

Countless  textbooks  describe  tools  and  techniques  for  decision  support 
but  do  not  convey  a  sense  of  how  decisions  are  made  or  how  analysis 
can  help.  Figure  3.2  suggests  that  the  outcomes  of  decisionmaking  are 
not  just  the  decisions  themselves  but  also  guidance  and  explanations — 
i.e.,  the  story.  The  guidance  is  necessary  for  implementation  and  expla¬ 
nations  are  needed  if  the  guidance  is  to  be  understood  and  the  deci¬ 
sions  accepted  by  the  President,  Congress,  and  diverse  stakeholders.2 

The  generic  model  described  in  Figure  3.2,  which  applies  specifi¬ 
cally  at  the  strategic  and  operational  levels  most  relevant  to  this  mono¬ 
graph,  recognizes  that  decisionmakers  are  usually  affected  by  consid¬ 
erations  other  than  formal  analysis  (see  the  items  on  the  left  and  right 
sides).  These  other  factors  reflect  their  own  backgrounds,  character¬ 
istics,  and  views  and  also  such  matters  as  concerns  about  the  indus¬ 
trial  base,  pressures  from  lobbyists,  or  their  administration’s  campaign 
promises.3  Thus,  analysis  informs  but  may  or  may  not  be  decisive  even 
if  well  done.  This  said,  analysis  can  be  more  helpful  if  it  considers  a  full 
range  of  appropriate  options,  considers  multiple  criteria  relevant  to  the 
decisionmaker,  confronts  uncertainty,  and  deals  well  with  the  effects  of 
disagreements  by  showing  alternative  perspectives.  To  elaborate  on  the 
latter,  senior  leaders  often  disagree  about,  say,  the  relative  importance 
of  modernization  and  current  readiness  or  about  the  relative  emphasis 
to  be  given  to  the  Asia-Pacific  or  Mideast  regions.  Although  it  may  be 
necessary  for  decisions  to  clearly  favor  one  perspective  or  another,  it 
is  common — and  consistent  with  the  FARness  principle — to  achieve 
“balance”  by  finding  options  that  account  “adequately”  for  the  full 
range  of  considerations  (assumptions,  uncertainties,  and  disagree¬ 
ments).  This  helps  both  to  build  consensus  and  to  hedge  against  uncer¬ 
tainty.  Overall,  Figure  3.2  is  saying  that: 
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Figure  3.2 
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•  Analysis  and  analysis  methods  should  include  diverse  options  and 
treat  criteria,  uncertain  information,  assumptions,  and  perspec¬ 
tives  as  variables — not  as  fixed  assumptions. 


Depth:  Balancing  the  Simple  and  Complex 

Most  of  what  follows  is  about  studies  and  analyses  that  require  depth 
of  inquiry.  Before  going  down  that  path,  however,  it  is  important  to 
recognize  that  only  some  analysis  is  of  that  character. 

Simplicity's  Dominance  in  More  Routine  Analysis 

As  those  familiar  with  such  organizations  are  quick  to  point  out,  sup¬ 
porting  decisions  in  a  high-level  analysis  shop  such  as  OSD  (Cost 
Assessment  and  Program  Evaluation)  or  the  related  offices  of  the  Joint 
Staff  and  services  often  requires  only  clear  and  logical  thinking,  arith¬ 
metic,  algebra,  and  skills  in  chart-making  and  writing  “one-pager” 
memos.  There  may  even  be  a  well-known  formula  that  does  a  “good- 
enough”  job.  For  example,  Navy  analysts  do  not  need  to  do  a  full  study 
to  understand  how  losing  some  aircraft  carrier  battle  groups  in  the 
budget  wars  would  affect  forward  presence  worldwide.  Army,  Marine 
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Corps,  and  Air  Force  analysts  do  not  need  a  new  study  to  estimate 
the  consequences  for  force  management  of  extending  units’  operational 
deployment  periods.  A  great  deal  of  analysis  on  the  margin  can  be 
rather  simple.  Analysis  often  depends  more  on  obtaining  credible  data 
(e.g.,  cost  estimates  or  geographic  information  relevant  to  deploying 
forces  or  targeting  adversaries)  than  on  mathematical  models.4 

With  this  said,  let  us  now  return  to  the  main  thrust  of  the  mono¬ 
graph,  which  is  about  issues  for  which  analysis  is  more  demanding. 

Achieving  Valid  Simplicity 

Figure  3.2  says  nothing  about  the  nature  of  the  analysis  itself,  but  it 
instead  reflects  a  recurrent  plea  from  policymakers  that  analysis  be 
simple  and  transparent.5  After  all,  they  must  ultimately  be  able  to  reason 
about  their  choices,  which  requires  intellectual  simplifications  (none  of 
us  can  “reason”  with  too  many  variables  simultaneously).  Further,  they 
must  communicate  to  and  convince  others.  And,  finally,  policymakers 
often  know  from  experience  that  the  most  important  issues  are  driven 
by  relatively  simple  considerations — if  one  can  merely  frame  the  issues 
properly  and  see  through  the  fog.  Finding  ways  to  see  through  the  fog 
has  long  been  a  prized  goal  of  system  analysts  and  policy  analysts. 

It  might  seem  that  analysis  and  underlying  models  should  there¬ 
fore  be  simple.  At  the  same  time,  however,  even  those  arguing  for  sim¬ 
plicity  want  the  “simple  story”  to  be  correct!  And  there,  of  course,  lies 
the  rub.  Can  these  considerations  be  reconciled?  The  answer  is  yes,  but 
it  means  that  simplicity  must  be  balanced  with  more  in-depth  work.  It 
is  wise  to  remember  the  last  phrase  of  a  good  version  of  Occam’s  Razor: 

Everything  should  be  made  as  simple  as  possible,  but  not  simpler. 

Often  ascribed  to  Albert  Einstein 6 

Ignoring  sound  but  contradictory  empirical  evidence,  then,  is  not 
acceptable.  So  also,  it  is  not  acceptable  to  disregard  the  implications  of 
logic,  experience,  and  credible  theory  extrapolating  beyond  what  has 
been  observed  so  far. 
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Reconciling  simplicity  and  sufficiency  of  depth  is  easier  to  appre¬ 
ciate  for  those  who  have  had  the  good  fortune  to  move  back  and  forth 
among  levels  of  detail.  A  physicist  or  chemist  is  expected  to  understand 
phenomena  at  levels  from  quantum  mechanics  through  thermodynam¬ 
ics  or  engineering  formulas.  Economists  are  expected  to  understand 
both  micro  and  macro  economics.  System  engineers  work  principally 
at  higher  levels  but  know  that  details  matter  when  the  system  modules 
are  connected. 

As  indicated  in  Figure  3.3,  telling  an  understandable  story  is  only 
one  reason  among  several  for  using  simple  models.  At  the  outset  of 
the  analysis  campaign  (the  bottom  of  the  figure),  one  may  ask  what 
seems  to  be  the  problem  and  what  a  “dynamite  result”  would  look 
like,  one  that  would  be  palpably  interesting  and  important — whether 
to  falsify  initial  notions  or  to  confirm  them.  Such  framing  can  focus 
analysis  and,  at  the  same  time,  motivate  research  with  the  depth  and 
integrity  to  test  competing  possibilities.  Moving  upward  in  Figure  3.3, 
relative  simplicity  enables  fast  and  agile  modeling  and  enables  “explor¬ 
atory  analysis”  across  the  vast  range  of  possibilities.  That  is,  it  enables 
parametric  modeling,  which  allows  one  to  see  forests,  not  trees,  and 
understand  major  tradeoffs.  Such  results  can  be  explained  clearly  and 
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persuasively.  Further,  the  key  parameters  of  such  work  become  natural 
metrics  following  logically  from  underlying  considerations7  rather  than 
being  ad  hoc  and  frequently  counterproductive. 

Simplicity  and  Complexity  in  Historical  Studies 

Arguing  for  a  combination  of  simplicity  and  underlying  depth  may  seem 
irregular,  but  a  casual  look  at  a  dozen  past  studies  suggests  otherwise. 
I  chose  major  studies  from  the  early  1960s  to  2010  with  which  I  was 
at  least  casually  familiar  and  in  some  cases  involved.  I  then  character¬ 
ized  the  studies  along  a  number  of  attributes  with  subjective  scores  of 
from  1  to  5,  with  5  indicating  a  heavy  dose  of  the  attribute.  (Appendix 
A  lists  the  studies  and  my  subjective  characterizations.)  For  example,  a 
study  with  a  5  in  the  “hard  physics”  category  might  have  involved  con¬ 
siderable  analysis  using  concepts  from  physics  and  technology.  On  the 
other  hand,  it  might  not  have  required  deep  thinking  about  concepts  of 
operations  and  would  get  a  score  of  1  or  2  in  that  category.  Figure  3.4 
summarizes  results  using  average  scores  across  the  12  studies  and  eight 
attribute  categories.  Readers  could  quarrel  with  the  choice  of  stud¬ 
ies,  the  attributes,  and  the  scoring:  The  results  are  merely  suggestive. 
Nonetheless,  the  studies  rather  consistently  included  a  mix  of  simple 
and  complex  modeling,  developing  concepts  of  operations,  doing  para¬ 
metric  capabilities  analysis,  and  performing  cost-effectiveness  analysis. 
That  is,  it  seems  that  in  relatively  large  studies,  analysts  have  found 
it  necessary  to  delve  deeply  into  problems  and  also  to  find  simplified 
ways  to  guide,  understand,  and  communicate  the  work.  Unfortunately, 
the  trend  over  the  last  decades  has  been  for  DoD  studies  to  become 
more  focused  on  standard  scenarios  and  big  models.  Restoring  a  better 
balance  should  be  an  imperative  for  those  who  manage  or  consume 
analysis. 

Another  trend  in  DoD  research,  but  not  yet  in  many  analytic 
studies,  has  been  recognition  of  social-science  considerations,  which 
were  largely  ignored  in  the  earlier  studies.  In  recent  times,  DoD  has 
been  experimenting  with  how  to  include  social  science  in  studies,  espe¬ 
cially  for  those  relevant  to  interventions  in  troubled  societies.8 
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Figure  3.4 

Attributes  of  Some  Past  Studies 
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Breadth:  Using  Diverse  Models  and  Tools 

Having  discussed  depth,  a  next  issue  is  how  to  broaden  analysis.  The 
need  to  do  so  is  even  more  evident  now  than  a  decade  or  so  ago  because 
social-science  factors  have  been  important  in  the  wars  in  Iraq  and 
Afghanistan.9  Good  decision-aiding  should  draw  on  a  mix  of  “hard” 
and  “soft”  information,  and  not  just  when  dealing  with  human  behav¬ 
ior.  Furthermore,  despite  the  emphasis  on  “rigor”  and  the  common 
fallacy  of  associating  rigor  with  “quantitative,”  many  important 
strategic-planning  questions  are  not  well  informed  by  purely  quantita¬ 
tive  analysis.  Examples  include  such  questions  as 

1.  How  can  the  United  States  best  deter  a  given  adversary?  To 
what  extent  should  deterrence  be  based  on  the  assumption  of 
“rational”  decisionmaking  and  how  can  we  understand  that 
rationality — in  both  peacetime  and  crisis? 
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2.  Is  deterrence  even  the  appropriate  focus  or  should  attention  be 
broadened  to  include  other  elements  of  influence,  such  as  co¬ 
optation,  dissuasion,  and  punishment  now  for  recent  actions  so 
as  to  deter  later  aggression? 

3.  How  well  should  we  expect  a  particular  strategy  or  related  tac¬ 
tics  to  work  when  they  interact  in  complex  ways  with  entire 
societies,  as  in  intervention  operations  with  the  intention  of  sta¬ 
bilization  and  reconstruction? 

4.  How  should  we  expect  our  adversaries  to  respond  and  adapt  to 
new  strategies,  capabilities,  or  actions  on  our  part? 

5.  What  does  history  tell  us  about  the  kinds  of  operation  that  we 
are  currently  considering? 

Such  questions  suggest  that  diverse  instruments  are  needed  in  an 
analysis  campaign.  Table  3.1  illustrates  this  with  instruments  in  rows 
and  columns  showing  their  strengths  and  weaknesses.10  Although  the 
ratings  are  subjective  and  depend  on  unspecified  assumptions,  the 
goodwilled  reader  will  probably  accept  the  rough  validity  of  Table  3.1 
and  acknowledge  that  different  insights  come  from  varied  models, 
human  war-gaming,  red-teaming,  and  historical  studies.  Some  instru¬ 
ments  are  good  for  narrow  decision  aids,  whereas  others  are  needed 
for  putting  the  pieces  together.  Campaign  models,  for  example — when 
used  with  large  negotiated  databases  for  only  some  standard  case — are 
poor  decision  aids  but  are  excellent  for  integration,  for  understanding 
the  many  facets  of  a  successful  large  operation,  and  for  building  analyst 
expertise  that  is  valuable  in  answering  specific  questions  quickly,  often 
with  simpler  models. 

Table  3.1  was  constructed  with  strategic-level  issues  in  mind, 
but  the  instruments  (rows)  would  be  different  if  the  figure  were  for  an 
analysis  campaign  at  the  mission  or  capability-class  level  (e.g.,  an  analy¬ 
sis  of  next-generation  armored  fighting  vehicles,  unmanned  submarine- 
launched  vehicles,  or  new  low-cost  fighter  aircraft  intended  for  partner 
militaries).  Also,  the  relative  importance  of  attributes  (columns)  would 
change.  There  would  be  less  interest  in  strategic  integration  (a  strength 
of  campaign  models)  and  more  interest  in  mission-level  models  and 
parameterized  engineering-level  models.  Instead  of  human  war- 


Table  3.1 

Broad  Scope  of  Analytical  Capabilities  to  Support  Decisionmaking 


Relative  Strength 


Instrument 

Simple  analytical3 

Seminar-level  human  war-gaming 

Red-teaming  on  capabilities  and  operations 

Qualitative  factor  trees 

Human  war-gaming 

Campaign  simulation  (usual) 

+  agents,  political-economic  factors,  and 
exploratory  analysis^ 

Mission-level  adaptive  models,  exploratory 
analysis 

High-fidelity  simulationc 
Historical  case  studies 
Historical  data  analysis 
Field  experiments  and  war  data 


Strategic-Level  Functionality 


Strategic 

Decision-  Strategic  Physical  Human  Empirical 

Resolution  Agility  Breadth  Aiding  Integration  Phenomena  Phenomena  Cautions 


NOTES:  Ratings  are  1  (very  poor)  to  5  (very  good),  with  red,  orange,  yellow,  light  green,  and  green  corresponding  to  1,  2,  3,  4,  and  5,  respectively. 
Scores  depend  on  assumptions. 

3Examples  include  closed-form  models  and  spreadsheet-level  computer  models. 

“Exploratory  analysis  examines  the  effect  of  simultaneous  variations  of  all  important  assumptions,  not  mere  sensitivity  analysis  on  the  margin. 
cln  some  instances,  high-fidelity  simulation  can  be  a  primary  and  reliable  source  of  what  can  be  considered  to  be  empirical  information.  It  is  simply 
not  feasible  to  obtain  the  equivalent  information  with  physical  testing. 
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gaming,  an  important  instrument  might  be  virtual  simulation  with 
human  operators  or  held  experiments. 

Regardless  of  the  level  of  the  decision  being  made,  the  supporting 
analytic  campaign  should  keep  in  mind  a  generic  rule  of  thumb: 

Analysts  working  primarily  at  a  given  level  of  detail  should  be 
familiar  also  with  representations  one  notch  deeper  in  resolution, 
to  ensure  an  adequately  deep  understanding  that  will  inform  their 
assumptions,  and  a  notch  lower  in  resolution  so  as  to  appreciate  the 
needs  of  decisionmakers. 

Yet  another  reason  for  breadth  is  that  it  is  often  possible  to  dis¬ 
cover  a  fruitful  way  ahead  by  examining  a  confusing  uncertain-laden 
problem  from  multiple  perspectives.11 

Why  doesn’t  everyone  use  a  range  of  instruments  in  their  analysis? 
Efforts  are  sometimes  made  to  do  so  in  DoD.12  Those,  however,  tend 
to  be  exceptions  because  of  the  influence  of  organizational  structure, 
culture,  experience,  management,  and  budgets.  People  with  different 
knowledge,  skills,  and  tools  tend  to  be  in  separate  organizations  with 
different  perspectives  on  what  constitutes  an  “analyst”  or  “analysis.” 
This  can  be  insidious  when  de  facto  organizational  doctrine  defines 
these  terms  narrowly  and  enforces  specialization.  The  best  analysts 
(and  consumers  of  analysis)  are  often  generalists  with  backgrounds  in 
such  subjects  as  the  physical  sciences,  engineering,  economics,  or  even 
political  science,  history,  and  law.  They  have  often  benefited  greatly 
from  applied  experiences,  to  include  commanding  or  planning  military 
operations.  Mathematics,  including  operations  research,  then,  is  only 
one  of  many  possible  backgrounds.  This  quickly  becomes  evident  on 
reviewing  the  backgrounds  of  past  giants,  as  conveyed,  for  example, 
by  the  oral  histories  of  senior  leaders  developed  by  the  Military  Opera¬ 
tions  Research  Society.  The  same  message  arises  from  discussions  with 
senior  industry  managers  about  their  hiring  practices,  even  for  model¬ 
ing-intensive  work. 
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Recognizing  Multiple  Criteria 

Good  analysis,  then,  should  balance  depth  and  breadth  and  use  appro¬ 
priately  diverse  tools.  But  what  should  analysis  address ?  One  approach, 
taken  in  some  DoD  documents,  is  to  organize  around  classes  of  risk 
(see  Appendix  C).  In  what  follows,  I  take  the  more  positive  approach  of 
organizing  around  objectives,  in  which  case,  risk  control  is  just  another 
kind  of  objective. 

Multiple  Objectives,  Including  Risk  Management 

One  eternally  challenging  aspect  of  defense  planning  is  that  it  is  so 
multifaceted  as  Figure  3.5  suggests  by  noting  that  DoD’s  planning 
must  address  different  arenas  worldwide  (including  the  global  arena 
and  functional  arenas  such  as  intercontinental  conflict,  space,  and 
cyberspace),  different  time  periods,  and  different  levels  of  challenge 
(e.g.,  the  spectrum  from,  say,  showing  occasional  military  presence  to 
being  capable  of  long-duration  conflict  with  a  capable  adversary).  This 
implies  multiple  criteria  for  assessment  of  force-planning  options — 

Figure  3.5 

Some  of  the  Many  Dimensions  of  Defense  Planning 
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criteria  that  cannot  sensibly  be  combined  or  measured  on  a  single  scale. 
After  all,  a  robust  plan  for  the  long  term  and  a  woefully  inadequate 
plan  for  the  short  term  and  mid  term  do  not  usefully  “average”  to 
“medium.”  The  Secretary  of  Defense  must  be  concerned  with  the  near 
term,  mid  term,  and  long  term  separately,  for  each  of  the  various  arenas 
and  levels  of  challenge.13 

Within  any  given  time  period  and  arena,  DoD  may  have  a 
number  of  objectives,  each  with  subobjectives,  sub-subobjectives  .  .  . 
down  to  discrete  tasks.  Assessing  how  well  the  capabilities  of  an  option 
are  “expected”  to  accomplish  each  of  these  constitutes  a  set  of  evalua¬ 
tion  criteria  (e.g.,  how  well  would  the  option’s  capabilities  do  in  a  best- 
estimate  scenario?).  A  top-level  force  planner  is  concerned  primarily 
with  an  aggregate-level  description.  Will  an  option  be  able  to  pro¬ 
tect  allies  from  invasion  or  coercion,  whether  by  deterrence  or  mili¬ 
tary  action  if  deterrence  fails?  Will  it  be  able  to  keep  the  sea  lanes 
open  to  commerce?  Such  higher-level  objectives  are  routinely  discussed 
in  national  planning  documents.14  The  next  level  of  detail  begins  to 
address  what  tangible  military  capabilities  are  needed  to  achieve  the 
various  objectives  and  subobjectives.  DoD  uses  “capabilities-based 
planning”  to  do  so,  as  discussed  below. 

A  relatively  specific  goal  such  as  being  able  to  defend  South  Korea 
from  invasion  or  coercion  by  North  Korea  leads  to  an  approximate 
hierarchy  of  objectives  and  capabilities.  Narrowing  the  same  example 
to  U.S.  and  South  Korean  air  forces  interdicting  an  invading  North 
Korean  army,  the  capabilities  needed  include  being  able  to  (1)  ensure 
presence  of  sufficient  aircraft,  (2)  quickly  achieve  high-quality  com¬ 
mand  and  control  (including  intelligence,  surveillance,  and  reconnais¬ 
sance),  (3)  suppress  or  evade  enemy  air  defenses,  and  (4)  attack  and 
destroy  fixed  and  moving  enemy  targets.  Such  a  “mission-system  per¬ 
spective”  recognizes,  as  shown  in  Figure  3.6,  that  all  four  of  these  are 
critical  components  for  success.  It  then  measures  the  extent  to  which 
the  defense  program  will  provide  all  the  capabilities  necessary.  Sup¬ 
pose,  for  example,  that  the  program  did  everything  except  provide  for 
suppression  of  air  defenses.  The  effectiveness  of  air  forces  against  an 
invading  army  with  air  defenses  would  then  be  very  low  during  the 
time  when  they  were  most  needed.15  That  is,  if  a  program  component 
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Figure  3.6 

Example:  Critical  Components  of  Interdiction  Capability 


Interdict  invading  army 
with  air  forces 


aircraft  and  command  and  control  enemy  air  defenses  maneuver  forces 
logistics  ar|d  ISR 

NOTE:  ISR  is  intelligence,  surveillance,  and  reconnaissance. 


is  critical,  then  a  program  that  ignores  it  is  fatally  flawed  even  if  it  the 
option  is  “efficient”  in  procuring  other  components.  Mathematically,  it 
follows  that  the  key  metric  for  a  related  program  would  be  nonlinear — 
in  this  case,  more  nearly  multiplicative  than  linear,  because  the  effec¬ 
tiveness  of  the  program  is  proportional  to  the  product  of  factors  indicat¬ 
ing  whether  each  critical  component  is  adequately  present.  More  of  one 
critical  capability  does  not  compensate  for  the  lack  of  another.16 

Analysts  with  this  perspective  will  relate  easily  to  and  be  seen  as 
useful  by  commanders  and  other  operators.  With  apologies  to  Moliere, 
we  can  imagine  a  commander  responding  to  a  briefing  on  such  matters: 

Ah,  you  mean  that  I  routinely  use  nonlinear  system  models  in  my 
thinking?  If  only  had  I  known!17 


Uncertainty  and  Disagreement 

If  options  are  to  be  evaluated  by  numerous  criteria,  then  what  about 
uncertainty  and  related  disagreements?  What  threats  may  materialize? 
Where?  When?  How  much  more  capable  might  they  be  than  expected? 
How  far  should  planning  go  to  hedge  against  the  risk  of  a  much-worse- 
than-expected  threat?  What  program  hedges  might  allow  for  cutbacks 
in  the  event  that  threats  develop  more  slowly  than  expected? 

Just  as  the  basic  evaluations  above  can  be  accomplished  using  test 
scenarios,  so  also  can  scenario  variants  represent  different  assumptions 
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about  threat,  circumstances,  and  the  future  generally.  Before  dealing 
with  that  matter  more  fully  below,  let  us  consider  the  problem  of  rea¬ 
soning  under  uncertainty  somewhat  more  abstractly.  How  much  can 
be  accomplished?  Based  on  a  combination  of  logic,  experience,  and 
historical  review,  the  most  feasible  and  productive  approach  (the  “sweet 
spot”)  is  arguably  as  follows: 

•  In  evaluating  an  option,  assess  (1)  how  well  it  is  expected  to  per¬ 
form,  (2)  the  potential  for  it  to  do  much  better  (upside  potential), 
and  (3)  the  risk18  that  it  will  do  much  more  poorly  (downside). 

•  In  each  of  these  assessments,  account  heuristically  for  both 
approximate  likelihoods  and  consequences.19 

This  may  seem  like  mere  common  sense,  but  it  is  in  contrast  with 
common  approaches.  It  is  not  mini-max  thinking  (planning  for  the 
worst  case);  it  is  not  rosy-scenario  planning;  it  is  not  thinking  restricted 
to  the  best  estimate.  Nor  is  it  overreaction  to  a  negative  shock  or  compla¬ 
cency  after  a  long  period  of  stability.  It  is,  however,  what  leaders  aspire 
to  when  they  are  working  hard  at  rational,  balanced  decisionmaking. 

Describing  this  analytic  sweet  spot  does  not  say  how  to  trade  off 
best-estimate,  most-favorable  (upside),  and  downside  (risky)  possibili¬ 
ties  in  reaching  a  decision.  Indeed,  no  such  rule  exists  for  the  most  con¬ 
sequential  decisions.  Yes,  one  can  always  refer  blithely  to  maximizing 
expected  subjective  utility  as  though  that  can  be  meaningfully  defined 
or  accomplished.20  However,  even  if  an  all-knowing  god  atop  Mount 
Olympus  could  know  everything  needed,  that  would  do  little  to  help 
real  people  affected  by  uncertainties,  misperceptions,  cognitive  biases, 
and  idiosyncratic  path-dependent  preferences.  Further,  people  know 
that  they  only  live  once,  so  that  “expected  value”  (over  many  lifetimes 
in  parallel  universes?)  may  have  little  salience.  Famous  historical  com¬ 
manders  have  sometimes  “gone  for  the  gold”  even  when  they  were 
aware  of  the  catastrophic  consequences  of  failure. 

It  seems  that  we  must  be  content  to  say  that  decisions  should 
somehow  “balance”  the  considerations,  a  term  favored  by  Secretary 
Robert  Gates21  and  used  heavily  in  RAND  work  on  strategy.22  In  prac- 
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tice,  national  leaders  concerned  about  those  for  whom  they  are  respon¬ 
sible  will  use  a  heuristic: 

•  Seek  first  to  avoid  disasters  by  not  even  considering  unduly  risky 
options. 

•  Within  such  constraints,  try  to  be  smart  (i.e.,  do  sensible  things 
and,  in  some  cases,  optimize). 

This  may  seem  easy,  but  consider,  for  example,  that — as  in  bargaining 
more  generally — leaders  may  need  to  use  bluff  and  bluster,  suggesting  a 
greater-than- actual  willingness  to  take  risks.  The  difficulty  of  doing  so 
both  credibly  and  responsibly  is  notoriously  difficult,  as  became  clear 
from  Cold  War  discussion  of  nuclear  matters23  and  the  recent  budget 
wars  in  Washington  that  led  to  sequestering  that  no  one  wanted. 

Defining  the  terms  of  this  sweet-spot  approach  is,  then,  by  no 
means  straightforward.  Optimists  may  underestimate  both  the  proba¬ 
bility  and  the  negative  consequences  of  the  “worst  case”  as  did  Saddam 
Hussein  in  both  1990  and  2003. 24  Our  own  leaders  also  make  mis¬ 
takes.  The  George  W.  Bush  administration  underestimated  the  down¬ 
side  risks  of  the  2003  war  with  Iraq — despite  notable  efforts  by  Secre¬ 
tary  of  Defense  Donald  Rumsfeld  to  identify  what  could  go  wrong  and 
demand  related  contingent  planning.25 

In  other  contexts,  political  leaders  may  miss  upside  opportunities. 
That  was  arguably  the  case  at  the  end  of  the  Cold  War  when  the  admin¬ 
istration  of  George  H.W.  Bush  was  skeptical  about  developments  in  the 
Soviet  Union.  It  soon  recognized  that  changes  were  real  and  engaged 
effectively,  but  some  histories  refer  to  1989  as  the  “Lost  Year”  and  spec¬ 
ulate  about  what  might  have  been.26  Some  observers  argue  that  the  Bill 
Clinton  administration  and  NATO  missed  an  opportunity  to  inter¬ 
vene  early  in  the  Bosnian  crisis  and  impose  stability,  which  might  have 
avoided  the  decade  of  misery  that  ended  with  the  Kosovo  operation. 
Clinton  himself  regarded  failure  to  stop  the  genocide  in  Rwanda  as  a 
great  error  (a  missed  opportunity  to  stop  massive  killing).  Another  con¬ 
sequential  example  arose  in  the  2000  Arab-Israeli  talks  when,  in  the 
dominant  narrative,  it  was  proven  again  that  “Arafat  has  never  missed 
an  opportunity  to  miss  an  opportunity.”27  Although  skepticism  is  war- 
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ranted,  some  accounts  argue  that  the  Bush  administration  failed  in 
the  early  2000s  to  recognize  a  window  of  opportunity  for  dealing  with 
North  Korean  leaders.28 

Even  though  the  sweet  spot  for  analysis  described  above  is  any¬ 
thing  but  simple,  it  is  a  sound  aspiration  for  framing  issues  and  for 
aiding  high-level  decisionmakers.  Analysis  can  help  a  great  deal  in 
doing  so. 

Measuring  Qualities  of  an  Option  Under  Uncertainty 

Given  distinctions  among  an  option’s  expected  performance,  upside 
potential,  and  risk,  how  do  we  measure  them?  The  most  common  way 
is  to  characterize  options  by  static  attributes  thought  to  correlate  with 
or  determine  both  performance  and  risk.  Examples  include  number 
of  brigades,  ships,  or  aircraft;  number  of  people;  network  bandwidth; 
percentage  of  budget  devoted  to  research  and  development  (seen  as 
reducing  future  risk);  and  quality  of  people.  Another  class  of  mea¬ 
sures  includes  such  items  as  number  of  carrier  battle  groups  on  station, 
number  of  wars  that  could  be  fought  simultaneously  or  at  least  concur¬ 
rently,  and  spin-up  time  for  some  functions. 

In  this  monograph,  I  do  not  address  such  attribute-based  mea¬ 
sures.  Instead,  I  focus  on  measures  typically  obtained  from  models  of 
one  sort  or  another,  usually  in  connection  with  scenarios. 

The  scenario  approach  has  been  widely  used  since  the  work  of 
Herman  Kahn  in  the  1950s.29  It  may  employ  thought  experiments  or 
use  models  to  estimate  the  performance  of  options  in  scenarios.  One 
scenario  might  correspond  to  a  somewhat  worse-than-expected  con¬ 
flict  (conservative,  but  not  strongly  so).  Other  scenarios  might  corre¬ 
spond  to  much  more  troublesome  conflicts  (e.g.,  short-warning  attacks, 
wars  in  which  allies  are  unwilling  to  cooperate,  or  wars  that  start  with 
major  degradation  of  U.S.  networks  or  space  systems).  Still  other  sce¬ 
narios  might  correspond  to  wars  in  which  circumstances  are  especially 
favorable  and  the  outcome  could  be  a  major,  favorable  change  in  a 
regional  balance  of  power  as  in  replacing  a  despotic  regime  with  a  well- 
supported  democratically  disposed  government,  as  some  envisaged  in 
2003. 
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Although  scenario  usage  is  ubiquitous,  the  art  and  science  of  how 
to  construct  suitable  scenarios  is  not.  This  merits  emphasis: 

•  A  fundamental  truth  in  analysis  is  that  scenarios  drive  the 
answers.  Thus,  much  effort  should  go  into  conceiving  and  tuning 
the  scenarios  used  and  specifying  uncertainty  ranges.  This  should 
be  a  deeply  analytic  affair  rather  than  the  result  merely  of  creative 
people  spinning  stories  that  raise  interesting  issues.30 

Military  staffs  of  the  various  services,  of  course,  quickly  recognize  how 
parameter  settings  in  DoD  scenarios  will  affect  their  services’  programs 
of  record.  They  advocate  accordingly.  From  a  national  perspective,  such 
parameter-settings  should  be  driven  by  explicit  strategic  considerations 
and  joint  analysis,  rather  than  by  de  facto  log-rolling  in  lower-level 
committees. 

Divergent  Thinking  and  Scenario  Space.  Figure  3.7,  drawn  from 
work  a  decade  or  so  ago,  indicates  the  concept  of  a  “scenario  space” 
(or  possibility  space).  The  left  side  shows  a  number  of  name-level  sce¬ 
narios  (i.e.,  scenarios  defined  only  to  the  extent  of  giving  them  a  name 
describing  the  particular  operational  challenge  that  they  pose).  The 
right  side  notes  that  specifying  what  any  given  name-level  scenario 
means  requires  thinking  about  a  multitude  of  variables  ranging  from 
political-military  context  to  the  various  actors’  strategies  and  objec¬ 
tives,  their  actual  capabilities,  and  so  on.31  To  think  about  test-case 
scenarios,  then,  should  mean  thinking  to  some  degree  about  the  full 
dimensionality  of  scenario  space. 

A  Process  of  Convergent  Thinking  with  a  Spanning  Set  of  Test 
Cases.  Musing  about  an  entire  scenario  space  is  a  heady,  abstract 
business.  For  down-to-earth  purposes  such  as  systematizing  analysis 
and  managing  change,  it  is  desirable  to  draw  insights  from  the  broad 
scenario-space  work  to  identify  a  smallish  discrete  set  of  test  cases, 
which  can  be  called  a  spanning  set,  such  that  a  force  able  to  handle 
the  test  cases  should  be  able  to  handle  real  cases  if  they  arise  in  parts 
of  scenario  space  that  are  of  interest.  These  are  perhaps  the  parts  that 
pose  significant  challenges  for  the  future  capabilities  but  that  are  also 
reasonably  plausible  (what  Kahn  sometimes  referred  to  as  “not  incred- 
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Figure  3.7 

The  Concept  of  Scenario  Space 
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ible”  to  suggest  the  realm  of  10  percent  likelihood  rather  an  0.1  percent 
or  50  percent)  and  about  which  it  is  possible  to  do  something.  It  is  con¬ 
ceivable  that  the  earth  will  be  hit  by  a  meteor,  but  defense  planners  can 
do  nothing  about  that.  The  situation  in  a  country  with  civil  war  may 
be  deplorable,  but  perhaps  the  United  States  cannot  do  anything  about 
it  and  might,  in  the  course  of  trying,  make  things  worse. 

Although  it  has  become  common  for  analysis  organizations  to 
focus  on  scenarios  established  by  policy  offices,  an  important  analysis 
role  is  to  help  identify  and  design  possible  planning  scenarios,  discuss 
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implications  with  policymakers,  and  subsequently  tune  the  scenarios 
so  that  they  accomplish  what  is  intended  in  the  rest  of  the  planning 
process.  My  own  conclusion  is  that 

•  This  function  should  be  accomplished  ultimately  in  a  partnership 
among  Office  of  the  Secretary  of  Defense  (OSD)  (Policy),  OSD 
(Cost  Assessment  and  Program  Evaluation),  the  Joint  Staff,  and 
OSD  (Acquisition,  Technology,  and  Logistics).* 

•  This  function  should  be  accomplished  with  a  small  group  of  ana¬ 
lysts  from  the  four  offices  working  together,  rather  than  with 
divided  responsibilities  and  sequential  processes.32 

The  concerns  of  the  services,  combatant  commanders,  and  defense 
agencies  would  be  represented  through  one  or  more  of  the  above  offices. 

Figure  3.8  describes  the  process  schematically.  Referring  to  the 
numbered  steps  of  the  process  of  Figure  3.8,  analysts  first  take  a  broad 
scenario-space  view  to  recognize  the  many  objectives,  constraints,  and 
uncertainties  (Step  1).  In  doing  so  they  confront  deep  uncertainties  and 
discover  how  the  scenario  space  breaks  into  regions  posing  different 
challenges  (Step  2).  For  a  given  region,  the  challenges  might  include, 
e.g.,  response  time,  technical  capabilities,  force  size,  and  troublesome 
adversary  strategies.  The  challenges  might  also  be  political,  economic, 
or  social,  as  in  “complex  endeavors”  generally  or  recent  wars  partic¬ 
ularly.  Next,  analysts  identify  representative  parameterized  scenarios 
for  each  challenge  region  (Step  3).  Given  suitable  models,  they  can 
do  first-cut  capabilities  analysis,  estimating  what  could  be  achieved  to 
meet  the  various  challenges  as  a  function  of  cost  (Step  4).  Since  cost 
and  feasibility  depend  on  the  stringency  of  challenges,  parameterizing 
stringency  becomes  part  of  the  analysis. 

The  process  next  envisions  going  to  policymakers  to  discuss  what 
capabilities  they  wish  to  pursue  further  given  results  of  the  first-cut 
analysis  (top  yellow  diamond).  Policymakers  then  make  initial  deci¬ 
sions,  giving  up  on  some  capabilities  but  pursuing  others.  That  is,  they 


These  represent  the  perspectives  of  strategy  and  policy,  program  analysis  and  suggesting 
choices  within  a  budget,  the  military  “operators,”  and  those  responsible  for  new  technology 
and  related  acquisition  (as  well  as  logistics). 
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Figure  3.8 

Moving  from  a  Scenario  Space  to  a  "Spanning  Set  of  Test  Cases" 
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decide  tentatively  on  “requirements.”  This  requirement-setting  must  be 
detailed  enough  to  define  intent.  Subsequently  (Step  5),  analysts  trans¬ 
late  qualitative  desires  (e.g.,  “deter,”  “achieve  an  early  halt,”  or  achieve 
“persistent  surveillance”)  into  parameter  settings  within  the  test-case 
scenarios  to  be  used  subsequently.  As  discussed  below,  parameter  set¬ 
tings  may  differ  for  different  functions. 

In  response  to  guidance,  the  services  and  defense  agencies  develop 
proposals  and  evaluate  them  against  the  test  cases  and  uncertainty 
ranges.  The  results  are  then  reviewed  and  analyzed  to  aid  decision¬ 
makers  in  making  “final”  decisions  on  forces,  weapon  systems,  etc. 
(Step  6).  Given  decisions,  analysts  then  adjust  prior  guidance,  test-case 
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scenarios,  metrics,  and  goals  accordingly.  The  process  continues  with 
further  iteration  occurring  in  subsequent  years. 

The  process  shown  would,  in  some  respects,  be  more  centralized 
in  OSD  and  the  Joint  Staff  than  the  current  process.  There  would  be 
fewer  “seats  at  the  table”  in  recognition  that  high-quality  decision- 
aiding  requires  coherence  and  that  the  offices  named  are  ultimately  the 
ones  responsible.  That  said,  the  services  and  defense  agencies  would  be 
the  source  of  much  information  and  advice,  the  analysis  could  be  as 
transparent  to  them  as  DoD  found  desirable,  and  the  sharing  of  infor¬ 
mation  and  tools  could  be  as  extensive  as  it  has  been  in  recent  years — 
something  widely  regarded  as  quite  valuable,  a  period  in  which  “open 
and  collaborative”  have  been  watchwords. 

A  key  output  of  Figure  3.8’s  process  concept  is  that  of  the  func¬ 
tion  of  test-case  scenarios  (Step  5).  These  could  fall  into  several  classes 
with  different  functions  in  the  overall  management  of  DoD:33 

1.  Cap  ability -development  tests  or  requirement-expressing  tests 
should  be  stringent,  demanding  as  much  of  the  options  as  is 
reasonable  given  the  nature  of  the  challenge  and  the  feasibility, 
under  a  budget,  of  developing  requisite  capabilities.  The  tests 
should  be  forcing  functions  of  changes  decided  on  by  leadership. 

2.  Representative  tests  should  test  against  what  might  be  seen  as 
reasonably  typical  or  even  “best-estimate”  versions  of  the  chal¬ 
lenges.  These  might  be  the  basis  for  education,  training,  normal 
exercises,  and  most  public  discussion. 

3.  Communication-related  tests  should  be  tailored  for  purposes 
of,  say,  summarizing  key  points  for  policymakers,  convincing 
Congress  why  a  capability  is  important,  and  explaining  broad 
intended  directions  to  the  organization  as  a  whole.  These  might 
be  a  subset  of  those  in  (1)  or  (2). 

4.  Operational-planning  tests.  At  any  given  time,  operations  plan¬ 
ners  need  to  build  plans  dependent  on  numerous  assumptions, 
although  hedged  where  possible.  Some  of  the  assumptions  are 
highly  sensitive  because  they  involve  undisclosed  and  possibly 
fragile  U.S.  capabilities  and  tactics,  intelligence  on  adversaries, 
or  judgments  about  likely  and  possible  behaviors  of  many  coun- 


Concrete  Suggestions  for  Analysis  37 


tries,  including  allies.  Such  assumptions  are  often  not  suitable 
for  use  in  (2)  or  (3)  above.  They  may  be  reflected  in  (1)  directly 
or  indirectly.  However,  some  may  be  ignored  for  good  reason. 
For  example,  if  the  United  States  today  has  a  highly  secret  and 
clever  countermeasure  against  an  air-defense  system,  capabil¬ 
ity  development  may  nonetheless  seek  the  ability  to  defeat  the 
air-defense  system  without  such  cleverness  because  the  special 
countermeasure  is  fragile.34 

The  differences  across  classes  of  test  case  are  not  always  as  one 
might  expect.  Historically,  operations  plans  have  included  assump¬ 
tions  that  were  more,  less,  or  equivalent  to  those  of  corresponding 
force-development  scenarios.  Also,  forcing  functions  of  change  (Class 
1)  have  sometimes  been  very  useful  for  communications  to  everyone 
from  Congress  to  the  broad  DoD  organization.  (See  Appendix  D  for  a 
discussion  of  Cold  War  examples.) 

Upon  reflection,  Figure  3.8  implies  an  iterative  approach  to 
analysis  in  which  a  small  set  of  analysts  does  the  first-order  analysis 
to  acquaint  decisionmakers  with  the  big  choices,  making  it  clear  what 
capabilities  can  plausibly  be  achieved  at  what  price,  but  only  roughly. 
With  such  information,  decisionmakers  can  expand  or  contract  their 
appetites  for  improvement,  after  which  analysts  can  sharpen  the  defini¬ 
tion  of  test  cases  so  that  they  become  “requirements”  for  the  organiza¬ 
tion  to  impose  subsequently.  This  has  often  been  the  way  that  decisions 
have  been  reached.  The  notion  that  objective  “requirements”  are  set  by 
the  combatant  commanders  with  Congress  and  DoD  then  providing 
the  necessary  resources  has  always  been  nonsense.  The  real  “require¬ 
ments”  are  what  the  policymakers  decide  to  demand  after  paying 
attention  to  strategy,  needs,  alternatives,  feasibility,  costs,  and  other 
considerations. 

Superficially,  it  may  seem  to  some  readers  that  this  is  already  the 
way  DoD  works,  since  senior  policymakers  discuss  and  agree  on  plan¬ 
ning  scenarios.  In  fact,  the  policymakers  often  do  not  understand  the 
implications  of  the  all-important  parameter  settings  in  these  scenarios 
that  emerge.  Sometimes,  they  do,  as  when  they  decide  to  require  abil¬ 
ity  for  near-si  mukaneous  conflicts  with  “near”  including  an  offset  of  a 
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specified  time.  Sometimes  they  do  not,  as  in  early  pronouncements  of 
intent  to  acquire  capabilities  to  halt  an  invasion  that  were  ambiguous 
about  how  quickly  that  needed  to  be  achieved.  Similarly,  early  calls 
to  achieve  persistent  surveillance  were  notoriously  ambiguous  about 
whether  that  meant  continuous  or  with  coverage  updates  of,  say,  min¬ 
utes,  hours,  or  days.35 


Exploratory  Analysis  for  Capabilities-Based  Planning 

Whether  for  broad  initial  work  or  more  pointed  analysis,  it  is  valu¬ 
able  to  have  capabilities  analysis  that  shows  results  as  a  function  of 
assumptions.  Figure  3.9  illustrates  this  by  contrasting  ways  to  compare 
two  options.  At  the  top  is  a  simple  bar-chart  summary  comparing  the 
options  for  a  standard  case.  Option  1  appears  somewhat  better.  The 

Figure  3.9 

Comparing  Options  Under  Uncertainty  (Point-Scenario  Analysis  Versus 
Capability  Analysis) 
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bottom  half  of  the  figure  is  in  the  spirit  of  capabilities  analysis,  preemp¬ 
tively  addressing  What-if?  questions  by  showing  results  as  a  function  of 
timeliness  and  mission  difficulty  (X  and  Y  axes,  respectively).  For  this, 
the  result  is  not  shown  on  the  Y  axis,  but  rather  by  the  location  within 
the  rectangular  space.  That  space  has  an  infinite  number  of  dots,  one 
for  each  scenario.  Only  one  such  dot  is  shown.  Results  for  a  given  sce¬ 
nario  are  good,  bad,  or  ambiguous  depending  on  which  colored  region 
it  sits  in. 

This  depiction  is  consistent  with  the  earlier  comparison  in  that 
both  Options  1  and  2  are  in  the  success  region  (green)  for  the  stan¬ 
dard  case.  However,  the  lower  display  shows  that  Option  2  is  superior 
overall  because  we  should  want  capabilities  that  ensure  good  or  at  least 
potentially  good  results  in  as  big  a  portion  of  the  space  as  possible.  In 
other  analysis,  results  might  depend  sensitively  on  more  than  two  key 
variables  and  such  comparisons  would  be  more  difficult  to  plot.  How¬ 
ever,  great  advances  have  been  made  in  such  matters  as  discussed.36 

As  a  second  example,  Table  3.2  compares  two  notional  options 
for  a  homeland  ballistic-missile  defense  system.  Option  2  costs  twice 
as  much  as  Option  1.  Because  ballistic-missile  defense  effectiveness 

Table  3.2 

Ballistic-Missile  Defense  Capabilities,  by  Threat  and  Objective  (Notional) 


Class  of  Threat 


Option  1  ($100B) 


Option  2  ($200B) 


Near-  Near- 

Minimum  Moderate  Perfect  Minimum  Moderate  Perfect 
Objective  Defense  Defense  Defense  Defense  Defense  Defense 


Massive  attack,  near 
peer 

Small  attack,  near  peer; 
or  multiple  missiles, 
advanced  rogue 

Multiple  missiles;  simple 
rogue 

Single  missile;  simple 
rogue 


SOURCE:  Davis,  Shaver,  and  Beck,  2008. 

NOTES:  Red  =  very  poor,  orange  =  poor,  yellow  =  medium,  light  green  =  good,  and  green  =  very 
good.  These  capabilities  might  be  quantified  in  terms  of  the  probability  of  intercepting  a  high 
fraction  of  attacking  missiles. 
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depends  on  the  nature  of  the  future  threat  (size,  simultaneity,  tech¬ 
nical  sophistication),  about  which  there  is  deep  uncertainty,  it  makes 
little  sense  to  assess  only  in  terms  of  a  standard  case.  Also,  it  is  impor¬ 
tant  to  distinguish  among  several  objectives  as  do  policymakers  them¬ 
selves:37  Having  some  defense  (minimum  defense)  is  very  desirable  to 
avoid  coercion  and  potential  self- deterrence — i.e.,  the  possibility  that 
if  a  rogue  state  were  attacking  a  U.S.  ally,  the  United  States  would  be 
deterred  from  responding  effectively  because  of  the  rogue’s  ability  to 
strike  the  United  States  with  a  nuclear  weapon.  Having  a  moderately 
effective  defense  that  would  intercept  most  of  an  attack  would  pre¬ 
sumably  further  improve  deterrence.  A  near-perfect  defense  would  be 
valuable  in  improving  deterrence  and  in  limiting  damage  if  deterrence 
failed.  Overall,  Option  2  provides  better  capabilities,  but  primarily  for 
modest  threats  without  advanced  counter  measures.  If  the  “standard 
case”  assumed  a  multiple-missile  attack  by  a  simple  rogue  (bottom 
row),  then  Option  2  would  be  superior.  In  the  larger  perspective,  how¬ 
ever,  it  would  be  a  more  difficult  judgment  call  to  choose  between 
options.  How  much  would  policymakers  be  willing  to  pay  for  a  system 
increasing  capability  only  against  a  primitive  threat?  There  might  be 
interest  in  a  better  Option  3,  even  if  considerably  more  expensive.  But 
would  that  be  technically  feasible  even  with  high  costs? 

Many  examples  might  be  given  of  capabilities-based  analysis 
summarizing  difficult  issues  comprehensibly  over  the  relevant  possibil¬ 
ity  space.  (Appendix  F  describes  exploratory  analysis  for  the  “halt  prob¬ 
lem,”  circa  2000.)  Modern  displays  help,  as  discussed  in  RAND  work 
on  robust  decisionmaking  and  by  analysts  from  the  United  Kingdom, 
Canada,  and  Australia  at  a  recent  meeting  of  the  Military  Operations 
Society38 

My  own  observation  here  is  that 

•  Policymakers  almost  always  prefer  the  kinds  of  uncertainty¬ 
conscious  results  that  come  from  capabilities-based  analysis — but 
only  if  it  is  comprehensible  and  helps  with  decisionmaking  (hand- 
wringing  about  uncertainty  is  not  helpful). 
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•  Managers  of  analysis,  however,  have  often  not  responded 
accordingly — bowing  instead  to  pressures  for  consensus  on  sce¬ 
narios,  detailed  instantiations  thereof,  and  models. 

A  major  reason  for  the  lack  of  response  by  analysis  managers  has  been 
the  desire  to  respond  to  earlier  policymaker  complaints  from  when 
senior  leaders  were  often  exposed  to  competing  analyses  that  could  not 
be  readily  compared  because  of  having  been  based  on  different  sce¬ 
narios,  data,  and  models.  This  was  frustrating  to  officials  because  they 
could  not  compare  “apples  with  apples”  and  to  the  services  because 
they  feared  that  analysis  was  not  a  “level  playing  field.”39  In  theory, 
the  solution  was  to  require  that  analyses  use  standard  cases  as  base¬ 
lines,  but  then  go  on  to  provide  additional  information.  Regrettably, 
the  “additional  information”  provided  has  often  been  meager.  This  has 
long  frustrated  managers  of  analysis  as  well  as  policymakers.40 


Endnotes 

1  As  one  example,  service  representatives  preparing  for  an  analysis  of  alterna¬ 
tives  draw  on  industry  knowledge  and  analysis  to  understand  what  can  actually 
be  developed,  with  what  risks.  They  and  DoD  may  also  reach  out  to  independent 
groups,  as  in  studies  of  prompt  conventional  global  strike  (National  Academy  of  Sci¬ 
ences,  2008).  As  a  second  example,  DoD  recently  had  a  comprehensive  study  done 
to  review  and  synthesize  social-science  knowledge  on  counterterrorism  (Davis  and 
Cragin,  2009). 

2  This  point  was  reinforced  by  the  comments  of  Vice  Admiral  (USN,  Retired) 
Stanley  Szemborki  at  a  May  2013  Military  Operations  Research  Society  meeting, 
based  on  his  experiences  as  Deputy  Director  of  OSD’s  Cost  Assessment  and  Pro¬ 
gram  Evaluation  office. 

3  Analysts  sometimes  despair  because  it  sometimes  seems  that  “political”  consider¬ 
ations  dominate.  Some  considerations  that  seem  “political,”  however,  are  strategic, 
but  in  a  larger  arena  (e.g.,  national  economics  or  environmental  policy).  Others 
reflect  legitimate  public  values,  whether  or  not  convenient  for  DoD.  That  said,  other 
decisions  are  indeed  purely  “political.”  Harold  Brown  describes  his  experiences  and 
thoughts  on  such  matters  in  a  recent  memoir  (Brown  and  Winslow,  2012). 

4  See  two  strategy-level  examples  (Johnson  et  ah,  2012;  Davis  et  al.,  2008a). 
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5  Senior  leaders  distrust  black-box  simulations  and  seek  improved  analytical  agility. 
This  was  discussed  at  some  length  in  a  Military  Operations  Research  Society  meet¬ 
ing  on  how  to  improve  what  was  then  called  the  Analytic  Agenda  (since  renamed  as 
Support  for  Strategic  Analysis)  (Sweetser,  2010). 

6  Whether  Einstein  actually  made  this  statement  is  uncertain.  It  is  consistent  with  his 
documented  statements,  but  more  pithy.  See,  for  example,  http://quoteinvestigator. 
com/2 0 11/05/1 3/einstein-simple/. 

7  An  example  for  the  Air  Force  discussed  improving  capabilities  for  close-air  sup¬ 
port  (Davis  et  ah,  2010).  Many  other  examples  exist,  such  as  the  approximation  for 
defendable  radius  show  in  the  illustrations  in  Chapter  Four,  where  a  natural  but 
nonintuitive  metric  depends  on  the  1/3  power  of  the  power- aperture  product  of  a 
radar  divided  by  the  target’s  cross  section.  Such  natural  metrics  are  conceptually 
similar  to  the  dimensionless  parameters  that  play  a  big  role  in  theoretical  chemistry 
and  physics. 

8  At  the  research  level,  see,  e.g.,  Davis  and  Cragin,  2009,  and  Fenstermacher  et 
ah,  2010.  Recent  work  in  support  of  operations  has  been  led  by  the  Marine  Corps 
Combat  Development  Command.  It  has  included  substantial  participation  by  social 
scientists  and  new  qualitative  methods  of  analysis. 

9  See  Petraeus  and  Amos,  2006,  and  McChrystal,  2013. 

10  The  depiction  iterates  on  earlier  depictions  (National  Research  Council,  2006; 
Davis  and  Flenninger,  2007). 

11  An  insightful  discussion  of  how  it  is  important  to  look  at  problems  from  different 
perspectives  is  Akst,  2009,  p.  418,  who  also  discussed  the  fact  that  most  analyses  are 
not  predictive  in  large  part  because  the  necessary  data  are  lacking. 

12  Even  if  the  paradigm  of  seeking  such  breadth  in  analysis  campaigns  has  been 
absent,  numerous  examples  of  seeking  greater  breadth  can  be  noted.  For  example, 
OSD  (Cost  Assessment  and  Program  Evaluation)  has  used  system-dynamic  models, 
human  gaming,  and  the  human-in-the-loop  simulation  Peace  Support  Operations 
Model  to  investigate  irregular-warfare  issues  (alluded  to  in  Luman,  2008,  p.  379), 
acknowledging  the  value  of  gaming  also  in  Military  Operations  Research  Society 
meetings  (Sweetser,  2010).  In  work  overseen  by  James  Bexfield  of  OSD  (Cost  Assess¬ 
ment  and  Program  Evaluation),  DoD  sponsored  substantial  research  on  the  social 
science  of  counterterrorism  and  irregular  warfare  at  RAND  (Davis  and  Cragin, 
2009).  In  recent  times,  the  Marine  Corps  Combat  Development  Command  has 
drawn  on  social  scientists  and  social-science  methods  rather  than  traditional  simu¬ 
lation  in  conducting  illustrative  analysis  for  complex  political-military-economic 
operations  (discussions  with  Michael  Bailey  and  Yuna  Wong).  In  earlier  years,  the 
Army’s  Concepts  Analysis  Agency  under  E.  B.  Vandiver  sponsored  numerous  his¬ 
torical  studies,  including  some  by  the  late  Trevor  Dupuy.  Many  other  examples  will 
be  found  in  the  Military  Operations  Research  Society  oral  history  series  of  leaders 
in  military  operations  research.  They  are  available  on  its  website:  www.MORS.org. 
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13  See  also  discussion  the  of  strategy  and  multiple  objectives  in  Kugler,  2006. 

14  Particularly  relevant  are  the  National  Security  Strategy  (Obama,  2010),  the 
strategic  guidance  for  the  Defense  Department  (Obama,  2012);  Defense  Planning 
Guidance,  which  may  be  classified;  Quadrennial  Defense  Review  (Gates,  2010); 
service-by-service  documents;  and  the  Chairman’s  National  Military  Strategy 
(Mullen,  2011). 

15  Assuring  that  all  critical  capabilities  are  planned  for  is  a  key  element  of  capabilities- 
based  planning  (Davis,  2002)  and  also  strategies-to-tasks  (better  called  objectives- 
to-tasks)  planning  (Kent  and  Simons,  1994). 

16  Heeding  the  implications  is  difficult,  as  can  be  seen  by  instances  of  failed  inter¬ 
national  interventions  for  which  failure  was  likely  because  of  problems  of  security, 
economics,  governance,  hatreds,  or  all  of  these  critical  factors  (Davis,  2011).  See  also 
the  discussion  by  Harold  Brown,  commenting  on  current  issues  (Brown  and  Win¬ 
slow,  2012). 

1  The  allusion  is  to  “Well,  what  do  you  known  about  that!  These  forty  years  now, 
I’ve  been  speaking  prose  without  knowing  it,”  a  comment  by  Monsieu  Jourdain  in 
Moliere’s  “The  Bourgeoise  Gentleman.” 

18  Here  “risk”  is  negative  as  in  natural  language.  In  finance  and  some  of  the  risk 
literature,  “risk”  is  often  seen  as  a  variance  usage  tracing  back  to  “Modern  Portfolio 
Theory”  (no  longer  modern),  when  computational  and  data  limitations  made  doing 
better  impractical.  William  Sharpe  and  others  later  made  distinctions  between  the 
good  and  bad  sides  of  variance  (Sharpe,  2006),  leading  to  “Post-Modern  Portfolio 
Theory.”  Markowitz  and  Sharpe  won  Nobel  Prizes. 

19  See  Davis,  Kulick,  and  Egner,  2005,  which  discusses  this  model  of  decision 
and  argues,  based  on  unpublished  research,  that  top  decisionmakers,  such  as  presi¬ 
dents,  are  doing  well  if  they  consider  uncertainty  even  to  the  extent  that  the  model 
describes.  Often,  upside  opportunity,  downside  risk,  or  both  are  given  short  shrift. 

20  The  inherent  limits  of  “bounded  rationality”  have  been  discussed  for  more  than 
a  half  century  by  such  notables  as  Nobelist  Herbert  Simon  (Simon,  1982). 

21  Gates,  2010. 

22  Davis  et  ah,  2008a;  Johnson  et  ah,  2012. 

23  See  Schelling,  1960,  1966,  and  Kahn,  1962,  1966. 

24  Saddam  Hussein  anticipated  boycott  and  perhaps  air  strikes  from  the  air  and 
sea,  far  less  than  what  ensued  (Brands  and  Palkki,  2012).  That  is,  he  recognized  but 
underestimated  risks.  After  the  earlier  2003  war,  Saddam  admitted  having  miscal¬ 
culated  President  Bush  and  his  intentions,  thinking  that  the  war  would  be  more 
like  the  shortened  air  campaign  of  the  Gulf  War  (60  Minutes,  2009).  Some  of  Sad- 
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dam’s  decisions,  particularly  the  decision  to  invade  Kuwait  in  1990,  were  driven  by 
nonrational  emotions  and  misperceptions  (Woods  and  Stout,  2010). 

25  See  Rumsfeld,  2006,  and  Feith,  2008,  the  latter  of  which  documents  many  key 
deliberations.  Rumsfeld’s  prewar  interviews  demonstrated  concern  about  downside 
possibilities  (Lehrer  Hour,  2003).  Actual  military  preparations  fell  short  for  Phase 
IV  (stabilization)  (Bensahel,  Oliker,  and  Peterson,  2009,  p.  l4ff;  Ricks,  2004).  Con¬ 
siderable  critical  thinking  went  into  early  Phase  IV  planning,  but  it  was  not  trans¬ 
lated  into  a  solid  operations  plan,  appropriate  staffing,  and  needed  preparations. 
When  the  time  came,  there  also  were  conflicts  about  who  was  in  charge  and  about 
higher-level  strategy  (Bensahel,  Oliker,  and  Peterson,  2009,  p.  40).  Overall,  military 
preparations  for  Phase  IV  were  driven  by  top-down  assumptions  that  proved  false: 
The  “worst  case”  was  much  worse  than  planned  for  (Bensahel,  Oliker,  and  Peterson, 
2009,  pp.  233ff). 

26  Hoffman,  2009. 

27  Ross,  2005.  The  expression  about  Arafat  is  credited  to  former  prime  minister 
Abba  Eban. 

28  This  argument  is  made  cautiously  in  a  little-known  book  (Chinoy,  2008)  based 
on  numerous  interviews. 

29  Herman  Kahn  developed  the  methods  at  RAND  in  the  1950s  and  then  his  own 
Hudson  Institute  (Kahn  and  Schelling,  2009).  Scenario  methods  have  subsequently 
been  used  extensively  in  the  commercial  world, 

30  Davis  and  Winnefeld,  1983;  Builder,  1983. 

31  The  scenario-space  concept  traces  to  the  1990s  (Davis,  1994b,  2002). 

32  Currently,  OSD  (Policy)  has  responsibility  for  writing  the  scenarios  in  a  high- 
level  description;  the  Joint  Staff  then  develops  associated  concepts  of  operations  and 
defines  most  of  the  details;  OSD  (Cost  Assessment  and  Program  Evaluation)  in  the 
past  oversaw  the  detailed  scenarios  in  simulations.  OSD  (Acquisition,  Technology, 
and  Logistics)  has  not  participated  extensively  in  this  part  of  the  process,  which  is 
part  of  the  “Support  for  Strategic  Analysis”  effort. 

33  Readers  with  long  memories  will  recall  that  distinctions  were  once  taught  to 
analysts  regarding  declaratory,  programming,  and  employment  policies.  What  poli¬ 
cymakers  said  publicly  was  one  thing;  what  was  actually  being  budgeted  for  was 
another;  and  what  was  planned  in  the  event  of  war  was  yet  another. 

34  Whether  to  count  on  countermeasures’  effectiveness  has  been  debated  for  at  least 
50  years,  as  with  debate  about  the  B-l  manned  bomber  (Quanbeck,  Wood,  and 
Thoron,  1976).  Analogous  debates  exist  today. 
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35  This  reflects  discussions  in  the  mid-2000s  with  Kenneth  Krieg,  then  Director  of 
Program  Analysis  and  Evaluation,  and  Michael  Wynne,  then  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics. 

36  RAN  D  work  on  uncertainty  in  national  defense  has  been  reviewed  recently  with 
extensive  pointers  to  contributing  literature  (Davis,  2012).  Another  report  illus¬ 
trates  how  n-dimensional  results  “project”  onto  two-dimensional  depictions  (Davis, 
Bankes,  and  Egner,  2007).  For  a  compilation  of  reports  using  robust  decisionmak¬ 
ing  methods,  see  http://www.rand.org/topics/robust-decision-making.html. 

37  The  distinctions  were  recognized  by  an  influential  commission  (Rumsfeld 
and  Commission,  1998).  Policymakers  were  quite  aware  of  the  limitations  of  the 
ground-based  midcourse  system  as  they  began  its  deployment  in  the  2000s.  They 
anticipated  that  improvements  would  occur  only  with  experience  and  time.  Even 
then,  as  discussed  by  independent  scientists,  capability  might  be  limited  because  of 
countermeasures  (Sessler  et  al.,  2000). 

38  See  Hoehl  and  Scales,  2011;  Taylor,  2013;  Solly,  2013,  and  a  brief  overview  of  the 
Military  Operations  Research  Society  conference  (Leonard,  Thomason,  and  Bex- 
field,  2013). 

39  Personal  communications  with  David  Chu  describing  issues  that  arose  when  he 
was  OSD’s  Director  of  Program  Analysis  and  Evaluation. 

4u  This  lament  has  been  made  to  me  by  numerous  OSD  senior  executives  who  over¬ 
see  analysis  and  by  policymakers  who  are  consumers  of  analysis. 


CHAPTER  FOUR 


Creating  Capability  Models 


Given  the  desire  for  “capabilities  analysis”  as  described  in  earlier 
chapters,  how  do  we  obtain  the  models  necessary  for  such  analysis? 
Although  the  best  approach  involves  a  range  of  analytic  methods  and 
models  of  varied  resolution,  as  described  in  Chapter  Three,  providing 
the  relatively  simple  models  needed  for  capabilities  analysis  is  a  special 
challenge — apparently  because  the  art  and  ethic  of  simple  modeling 
has  been  lost  to  a  considerable  degree  as  newer  generations  of  analysts 
have  grown  up  using  complicated  computer  models.  Thus,  this  chapter 
deals  largely  with  how  to  obtain  the  lower-resolution  capability  models. 


Definitions  and  Examples 

Definition 

As  a  preface,  it  is  appropriate  to  discuss  what  is  meant  by  “capability 
model”  in  the  context  of  this  monograph. 

•  A  capability  model  is  a  causal  model  that  allows  us  to  under¬ 
stand  degree  of  mission  success  as  a  function  of  input  variables 
expressed  parametrically. 

To  simplify,  suppose  that  we  had  a  mission,  the  success  of  which 
was  measured  by  Y.  Suppose  that  Y  depends  on  X,  Y,  Z,  and  Q.  If  we 
had  a  black-box  model  that  allowed  us  to  input  the  values  of  X,  Y,  Z, 
and  Q,  at  which  point  it  would  return  a  value  of  Y,  that  model  would 
be  doing  point-case  analysis.  What  we  want,  however,  is  something 
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like  Figure  4.1,  which  shows  how  degree  of  success  changes  as  X,  Y, 
Z,  and  Q  are  varied  simultaneously.  In  some  contexts,  we  could  say 
that  our  capability  to  do  the  mission  adequately  depends  on  having  a 
combination  of  X,  Y,  Z,  and  Q  such  that  results  are  above  the  dashed 
line.  Note  how  easy  it  is  to  gain  insights  from  the  graph.  For  example, 
observe  that  if  Q=  Qj,  then  success  can  be  achieved  with  much  smaller 
values  of  X  than  if  Q  =  Q,,  but  success  can  be  obtained  for  either  value 
of  Q. 

This  type  of  display,  familiar  to  scientists  and  engineers,  is  ideal 
for  discussing  consequences  of  uncertainty.  For  the  example,  if  we 
know  that  X  is  in  the  range  shown  by  the  dotted  lines  and  Z’s  value 
is  between  Z2  and  Z .  and  that  Q’s  value  is  Q,,  then  the  uncertainty 
in  Y  (for  Z  =  Z,)  is  shown  by  the  horizontal  dotted  lines:  The  mission 
is  barely  successful  if  Z  =  Z3  and  fails  if  Z  =  Z^  or  Zr  In  contrast,  if 
Q=  Qj,  then  a  good  result  is  ensured  for  any  value  of  Z.  Note  that  while 
this  display  is  continuous,  it  is  in  the  same  spirit  as  the  capabilities- 
analysis  examples  of  the  last  chapter. 

This  type  of  chart  might  be  generated  by  running  a  “big  model” 
many,  many  times  while  holding  a  number  (perhaps  hundreds)  of  vari- 

Figure  4.1 

Schematic  of  a  Parametric  Capability  Model's  Output 
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ables  constant,  but  what  if  results  depend  also  on  those  inputs — as  in 
fact  they  often  do?  A  good  “capability  model”  should  not  have  impor¬ 
tant  hidden  dependencies. 

A  capability  model  may  take  the  shape  of  a  formula,  perhaps 
exploiting  equilibrium,  steady-state,  or  boundary  conditions  for  suc¬ 
cess.  Alternatively,  it  may  take  the  shape  of  a  time-dependent  model 
(closed-form  or,  more  typically,  a  computer  simulation)  that  is  struc¬ 
tured  like  the  operation  necessary  to  accomplish  the  mission.  It  identi¬ 
fies  the  critical  capabilities  for  success  and  represents  their  interaction, 
but  in  a  relatively  low-resolution  way.1 

Defining  the  concept  in  more  detail  is  not  so  simple  because: 

1.  Not  all  “formula  models”  are  simple,  and  most  simple  models  of 
interest  are  not  formula  models. 

2.  “Simple”  and  “spreadsheet”  are  not  synonyms:  Many  spread¬ 
sheet  models  are  treacherous  and  many  simple  models  are  not 
implemented  in  spreadsheets. 

3.  Some  simple  models  use  only  elementary  physics,  algebra,  or 
economics,  but  others  require  some  knowledge  of  calculus,  dif¬ 
ferential  equations,  or  linear  algebra.  How  “simple”  a  model  is 
depends  in  part  on  one’s  background. 

4.  Some  models  are  static  or  describe  a  steady-state,  while  others 
are  structured  like  operations  and  lend  themselves  to  simulation 
(e.g.,  as  with  successive  steps  of  a  kill-chain).  These  are  simple  as 
they  are  fairly  aggregated  (perhaps  using  “average  shooters”  or 
average  probabilities  of  detection),  but  are  complex  enough  to 
represent  all  critical  components  of  an  operation.2 

5.  Some  models  are  simple  by  virtue  of  taking  a  very  narrow  view, 
while  other  models  are  comprehensive  but  use  low-resolution 
variables. 

Figure  4.2  illustrates  the  last  point  schematically.  Suppose  that  the 
complete  model  (everything  within  the  outer  rectangle)  has  input  vari¬ 
ables  X,,X,,X,,  Y,,  Y_,  Y,,  Z,,  Z_,  Z„,  O, ,  O.,  O,  intermediate  variables 
X,  Y,  and  Z;  and  outputs,  Overall  Effects.  One  simple  model  would 
be  the  one  shown  in  dark  blue  on  the  middle  left.  It  would  focus  only 
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Figure  4.2 

Types  of  "Simple  Submodels" 


Full  model:  - - 

broad  and  deep  ' 
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on  understanding  an  intermediate  variable  X  that  might  be  one  of  the 
important  effects.  This  simple  model  would  haveX^X,,  andX3  inputs, 
and  X  as  its  output.  It  would  ignore  all  other  considerations.  A  different 
kind  of  simple  model  would  be  the  one  in  the  lighter  blue  region  (top 
center).  It  would  attempt  to  explain  overall  effects  but  would  do  so  by 
using  aggregate  variables  X,  Y,  and  Z  an  inputs  rather  than  the  larger 
number  of  lower-level  variables.  This  second  model  would  be  compre¬ 
hensive,  but  not  as  detailed. 

Examples 

What,  then,  is  simple?  What  matters  primarily  is  the  number  of  input 
variables  (not  counting  data  inputs  that  are  constant  and  known  accu¬ 
rately).  Simple  models  will  have  perhaps  three  to  ten,  rather  than 
dozens,  hundreds,  or  thousands  of  inputs.  To  early  system  analysts  and 
even  to  today’s  design  engineers  and  many  hard  scientists,  such  models 


Creating  Capability  Models  51 


are  familiar  and  routine.  To  some  modeling  and  simulation  shops, 
however,  they  may  be  rather  foreign. 

Below,  I  give  diverse  examples  of  simple  models.  As  implemented, 
they  may  be  part  of  relatively  simple  computer  programs,  but  for  dis¬ 
cussion  purposes  key  formulas  are  shown.  Briefly,  for  each  item: 


1.  Lancbester  equations  are  widely  used  in  classrooms  and  in  some 
analysis,  usually  to  illustrate  certain  concepts  approximately. 
Closed-form  solutions  are  possible  when  the  coefficients  are 
constant.3  I  show  them  first  because  of  their  familiarity,  but 
their  importance  is  often  exaggerated. 


dA 

dt 


k  n  dD 

-KdD,— 


-KaA 


where  A,  D,  and  the  Ks  represent  attacker  and  defender  force  levels 
and  killing  rates. 

2.  Layering  models  of  ballistic-missile  defense  are  widely  used  in 
studies  of  defense  systems.  If  layers  of  defense  are  independent, 
multiple  layers  can  greatly  improve  defense  effectiveness.4 

Pk=l-(1-P1)(l-P2)(l-P3) 

where  Pk  is  the  probability  of  intercept  given  that  Pp  P2,  and  P3  are 
independent  probabilities  for  intercepts  in  successive  layers. 

3 .  Defendable  radius  of  a  defense  estimates  whether  a  defense  system 
has  the  kinematic  potential  for  being  effective  (deferring  con¬ 
sideration  of  measure-countermeasure  phenomena,  interceptor 
and  warhead  quality,  and  so  on).  For  such  potential,  the  radar 
must  detect  the  target  far  enough  away  so  that  it  can  launch  an 
interceptor  that  will  reach  the  incoming  missile  at  a  significant 
range  from  the  target.  The  equation  proved  very  useful  in  early 
arms-control  analysis  (could  surface-to-air  missiles  be  upgraded 
to  have  anti-ballistic-missile  capability?)  and  subsequently  in 
analysis  of  whether  advanced  radars  later  associated  with  Aegis 
and  Patriot  had  at  least  some  ballistic-missile  defense  poten- 
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tial.5  This  does  not  describe  dynamics,  but  rather  the  conditions 
under  which  intercept  can  occur. 


_  Vi  f  c 
Vi+Vmj(PAa1/3 


VmA 


where  R  is  the  distance  at  which  the  missile  is  killed 
Vj  and  Vm  are  the  speeds  of  the  interceptor  and  missile 
P  is  radar  power 
A  is  radar  aperture 

C  is  an  empirical  constant  for  the  radar 
G  is  the  radar  cross  sections 

A  is  the  time  delay  between  detection  and  launch  of  the 
interceptor. 

4.  Halt  models  were  used  to  understand  the  potential  for  long- 
range  fires  (whether  from  aircraft  or  artillery)  to  interdict  and 
stop  invasions  of  road-bound  mechanized  forces.6  Both  closed- 
form  and  simulation  models  have  been  used. 

_  -KSA0  ±  ^/(KSA0)2  +  2KSA0 
h  KSR 

The  halt  time  is  given  as  a  function  of  kills  per  shot,  shots  per 
day,  initial  number  of  shooters,  and  the  deployment  rate  of 
shooters.  Halt  distance  is  the  product  of  halt  time  and  the 
movement  rate.  See  Appendix  F. 

5.  Multiresolution  precision-fire  modeling  have  been  used  to  under¬ 
stand  how  the  effectiveness  of  precision  fires  against  maneuver 
forces  depends  on  not  just  weapon  characteristics,  but  also  char¬ 
acteristics  of  terrain;  maneuver  tactics;  and  intelligence,  surveil¬ 
lance,  and  reconnaissance.7  Both  closed-form  and  simulation 
models  have  been  used  with  selective  detail  (Figure  4.3). 
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Figure  4.3 

Selective  Detail  in  a  Simple  Precision-Fires  Model 
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The  applicable  equation  is  not  aesthetic  to  read,  but  the  diagram 
suggests  its  simplicity. 

6.  Factor  tree  models  are  qualitative  models  showing  the  factors 
causing  an  effect  and  which  combinations  of  them  are  necessary 
for  the  effect  to  occur.  Given  social-science  uncertainties  and 
the  shortcomings  of  data,  such  models  can  sometimes  convey 
a  sizable  percentage  of  the  available  knowledge.  They  are  valu¬ 
able  for  integrative  discussions,  for  identifying  potential  lines  of 
action,  and  for  noting  where  particular  actions  will  likely  fail 
because  of  substitution  effects  (e.g.,  enemy  adaptation)  (Figure 
4.4). 8 


Figure  4.4 

Qualitative  Factor-Tree  Model  of  Public  Support  for  Terrorism 
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NOTES:  Applies  at  a  snapshot  in  time.  Current  factor  values  can  affect  future  values  of  some  or  all  other  factors. 
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7.  Uncertainty  sensitive  computational  versions  of  factor-tree  models 
make  factor-trees  quantitative  but  are  used  for  exploratory  anal¬ 
ysis  rather  than  point  predictions.  Visual  programming  can 
reveal  structure  and  ensure  that  it  matches  that  of  the  original 
factor  tree  (Figure  4.5).9 

8.  Empirical  models  (e.g.,  simple  regressions  inferred  from  experi¬ 
ence)  can  sometimes  dramatize  the  potential  for  failure  unless, 
for  some  reason,  this  time  truly  is  very  different  from  previ¬ 
ous  times.  The  particular  chart  represents  analysis  before  the 
Iraq  war  that  warned  about  many  more  “boots  on  the  ground” 
having  been  historically  employed  in  successful  stabilization 
operations  (Figure  4.6).10 

Illustrative  Simple  Models  Used  in  Defense  Studies 

The  intent  of  these  examples,  then,  is  to  illustrate  that  the  nature  of 
“simple  models”  varies  and  that  such  models  can  provide  comprehen¬ 
sible  insights  and  valuable  parametrics  in  a  wide  variety  of  problems. 
Those  who  focus  exclusively  on  computations  with  large  computer 
models  might  bear  these  in  mind. 


Building  Capability  Models 

Low-Resolution  Capability  Models 

The  reader  convinced  by  the  virtues  of  simplicity  and  analysis  under 
uncertainty  may  reasonably  ask  how  to  build  suitable  models.  Unfor¬ 
tunately,  there  is  no  school  solution.  By  and  large,  anyone  with  a  solid 
education  in  the  physical  sciences  or  engineering  is  likely  to  have  the 
core  skills,  if  not  necessarily  the  knack.  The  examples  listed  above 
convey  common  characteristics  of  such  models.  In  particular,  the  key 
equations  are  small,  with  perhaps  2-12  parameters  rather  than  dozens 
or  hundreds.  The  key  variables  are  represented  as  variable  parameters 
rather  than  as  fixed  “data”  (i.e.,  constants).  The  models  may  involve 
simple  physics,  less-simple  physics  or  engineering,  simple  probabil¬ 
ity  theory,  and  elementary  differential  and  integral  calculus.  All  can 
be  implemented  in  spreadsheet  programs,  such  as  Microsoft  Excel®, 


Figure  4.5 

Computational  Version  of  the  Factor  Tree 
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Figure  4.6 

Empirical  Model  of  Ground-Force  Levels  in  Conflicts 
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although  other  implementations  are  often  superior  for  one  reason  or 
another. 

As  with  analysis,  some  people  are  good  at  higher-level  modeling; 
others  are  not.11  Talent  and  background,  for  example,  are  necessary 
when  modeling  how  concepts  of  operations  should  change  as  context 
and  capabilities  change.12  Such  automated  adjustment  is  essential  if  one 
is  going  to  use  a  model  for  exploratory  analysis.  Also,  a  good  deal  of 
judgment  is  necessary  in  deciding  what  to  include  and  not  include,  how 
to  express  assumptions  and  uncertainties,  and  how  to  deal  with  diverse 
cases  and  avoid  misuse.  One  example  of  misuse  may  suffice.  Modern 
modeling  software  makes  probabilistic  and  statistical  calculations  easy. 
Even  someone  with  primitive  knowledge  can  build  a  program  to  run 
calculations  and  produce  seemingly  impressive  charts,  referring  pre¬ 
tentiously  to  having  done  “Monte  Carlo  calculations.”  They  may  not 
even  be  aware  that  the  software  builds  in  the  assumption  that  the  vari¬ 
ables  are  probabilistically  independent.  Nonetheless,  they  show  results 
comparing  the  probabilities  of  success  for  different  options  to  multiple 
significant  figures.  A  more  competent  analyst/modeler  might  get  dras- 
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tically  different  results  by  taking  into  account  correlations.  If  an  adver¬ 
sary  achieves  operational  surprise,  for  example,  he  might  cause  damage 
to  the  U.S.  network  and  disrupt  some  space-based  surveillance,  and 
strike  when  U.S.  forces  are  preoccupied.  These  variables  may  be  math¬ 
ematically  independent  (legitimate  independent  variables),  but  their 
real-world  values  are  not  statistically  independent. 

Ultimately,  then,  the  admonition  is  to  put  good  people  on  the  job, 
ensure  that  they  have  backgrounds  in  substantive  modeling,  not  just 
mathematics,  and  ensure  that  the  model  designs  and  implementations 
are  well  reviewed.  Such  an  admonition  may  seem  banal,  but  experience 
suggests  that  it  is  not. 

Motivated  Meta  Models 

An  alternative  way  to  generate  a  simple  parametric  model  is  to  run 
computational  experiments  with  a  well-validated  computer  model  and 
a  good  experimental  design.  The  trusted  model  can  be  run  many  times 
to  generate  a  “metamodel”  or  “response  surface”  for  subsequent  para¬ 
metric  analysis.  The  most  common  version  of  this  approach,  as  taught 
in  classrooms,  is  to  use  standard  statistical  methods.  The  output  is  a 
regression  equation  that  “explains”  the  results  of  the  full-up  simulation 
with  only  a  handful  of  variables.  If  the  correlations  are  strong  enough, 
the  regression  might  then  be  used  to  extrapolate  to  other  cases. 

This  approach,  regrettably,  has  what  I  see  as  fatal  flaws.  First,  the 
result  is  not  typically  an  intuitive  physical  model  in  terms  of  which  to 
reason  and  debate,  but  rather  a  regression.  Second,  the  response  sur¬ 
face’s  variables  may  not  be  the  appropriate  ones  on  which  to  focus. 
Third,  the  “dirty  little  secret”  of  supposedly  theory-agnostic  statisti¬ 
cal  analysis  is  that  it  is  assuming  one  or  another  mathematical  form, 
such  as  linear  weighted  sums,  when  the  phenomenon  may  be  mani¬ 
festly  nonlinear,  in  which  case  a  linear  metamodel  may  be  downright 
counterproductive.13 

A  better  approach  is  motivated  metamodeling  in  which  statisti¬ 
cal  apparatus  is  used  to  test  a  postulated  simple  model,  rather  than 
to  find  coefficients  of  a  regression.11  Although  I  introduced  the  name 
“motivated  metamodeling,”  the  concept  is  much  the  same  as  when 
physical  scientists  use  experimental  data  to  see  if  their  best  theoreti- 
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cal  model  does  a  good  job  or  needs  correction  terms.  If  the  postulated 
simple  model  is  well  conceived,  then  its  variables  will  be  the  “right 
ones”  for  discussion  and  the  form  of  the  model  will  itself  be  explain¬ 
able.  Predictions  of  the  motivated  model  can  be  discussed  as  intuitively 
understandable  results  plus  some  modest  correction  terms  that  reflect 
complicated  details.15  Such  a  model  has  profound  advantages  for  data 
analysis  from  computational  experiments  because  it  makes  available 
to  the  computer  the  composite  variables  (aggregation  fragments)  that 
are  more  likely  than  lower-level  variables  to  have  clear  causal  signifi¬ 
cance.  Suppose,  for  example,  that  an  effect  depends  on  the  product  of 
a  speed  V  and  a  time  T.  If  the  computer  uses  linear  regression  to  find 
a  metamodel,  it  may  find  that  the  coefficients  of  V  and  T  are  large.  It 
will  not,  however,  discover  that  the  product  of  V  and  T  is  crucial  unless 
given  the  “hint”  to  look  for  that  product. 

Lest  this  seem  too  abstract,  consider  again  the  models  shown 
above,  which  had  several  examples  of  formulas  capturing  essential 
features  of  the  problems  for  which  they  were  derived.  Except  for  the 
Lanchester  equations,  which  have  a  poor  basis  in  theory,  the  simple 
models  are  all  candidates  to  inform  motivated  modeling.  In  some 
analysis,  such  as  effectiveness  of  precision  fires  for  attacking  columns 
moving  through  mixed  terrain,16  interdiction  of  mechanized  armies 
on  roads,17  and  epidemiological  modeling  of  terrorism,18  my  colleagues 
and  I  have  had  good  success  with  motivated  metamodeling.  More 
important,  something  akin  to  this  approach  has  been  used  for  many 
years  in  the  physical  sciences. 

The  reader  may  at  this  point  say  “Ok,  I’m  convinced,  send  me 
the  people  who  can  do  the  simple  modeling,  whether  for  standalone 
modeling  or  the  basis  of  a  motivated  metamodel.  Where  are  they?”  I 
have  been  told,  by  at  least  two  directors  of  government  analysis  orga¬ 
nizations,  that  their  personnel  are  often  not  good  as  such  things — 
especially  if  they  are  essentially  mathematicians,  computer  scientists, 
or  programmers.  Nonetheless,  many  people — not  just  legends  from 
bygone  eras — have  the  requisite  talents.  Much  of  what  one  learns  in 
the  physical  sciences,  engineering,  biophysics,  and  applied  mathemat¬ 
ics  (especially  if  operations-oriented)  amounts  to  how  to  think  in  terms 
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of  relatively  simple  physical  models.  Finding  the  right  people  is  a  chal¬ 
lenge  for  any  organization,  but  supply  should  not  be  the  problem. 

I  should  emphasize  that  in  some  domains  with  complicated  phe¬ 
nomena  and  abundant  data,  more  inductive  statistics-driven  approaches 
can  have  great  value  and  even  be  pragmatically  superior  to  causal  mod¬ 
eling.19  That  said,  mindless  statistical  analysis  seems  to  me  to  be  a  big 
problem  in  military  analysis.20 

Complex  Models  That  Can  Support  Capability  Analysis 

A  theme  through  much  of  this  monograph  is  the  virtue  of  simple 
models.  Simplicity,  however,  is  an  elastic  and  subjective  concept. 
Campaign  models,  for  example,  are  often  seen  as  large,  complex,  and 
opaque — but  not  by  their  developers  and  expert  users.  They  are  not 
necessarily  opaque,  and  they  can  be  made  suitable  for  the  kind  of  para¬ 
metric  exploratory  analysis  in  support  of  higher-level  decisionmaking 
described  in  this  monograph.  Experience  suggests  to  me  that  this  will 
typically  not  happen,  and  that  analytic  organizations  would  do  better 
to  build  separate  simple  models  if  they  need  such  work.  However,  that 
is  a  commentary  on  organizations,  processes,  and  management,  not 
a  technical  conclusion.  In  fact,  “large”  models  can  have  many  of  the 
virtues  of  smaller  models,  and  additional  virtues  as  well.  The  keys  are 

•  quality  of  design,  to  include 

-  representation 

-  modularity 

-  selected  multiresolution  modeling,  for  the  core  model,  displays, 
or  both 

•  management. 

The  representation  chosen  determines  whether  the  model  relates 
well  to  the  most  important  entities  and  processes.  Modularity  allows 
one  to  “turn  off”  aspects  of  the  model  that  are  unnecessary  or  inappro¬ 
priately  burdensome  for  a  particular  analysis.  This  also  leads  to  a  cor¬ 
respondingly  simplified  interface  to  work  with.  Multiresolution  model¬ 
ing  refers  to  building  in  some  alternative  submodules  with  varied  levels 
of  detail  so  that,  for  example,  one  may  “turn  off  details”  and  use  a 
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simplified  representation  of  some  phenomenon,  again  with  a  suitably 
simplified  interface.  The  word  “selected”  appears  because  such  multi¬ 
resolution  modeling  features  are  practical  for  only  some  model  func¬ 
tions.  “Management”  in  this  context  refers  to  insisting  on  such  features 
in  the  design  phase,  assuring  that  they  are  maintained  subsequently, 
developing  and  maintaining  simplified  databases,  exercising  the  capa¬ 
bilities,  allowing  competition  by  testing  alternative  modules  as  they  are 
proposed,  and  resisting  pressures  to  “enrich”  the  model  in  ways  that 
make  it  less  modular  and  more  monolithic  without  preserving  the  fall¬ 
back  option  of  simpler  representations. 

As  an  example  based  on  personal  experience,  the  original  Joint 
Integrated  Contingency  Model  (part  of  the  1980s  RAND  Strategy 
Assessment  System,  RSAS)  was  modular.  This  permitted  studies  in 
which  only  the  ground-combat  model  was  used  (sometimes  with  an 
overlay  representing  certain  air-to-ground  effects).  Doing  so  was  rela¬ 
tively  easy  and  the  user  did  not  need  to  worry  about  the  hundreds  or 
thousands  of  other  input  variables.  Modularity  also  paid  off  later  when 
the  original  algorithms  for  assessing  ground-combat  attrition  were 
optionally  replaced  by  a  killer-victim-scoreboard  approach.  The  origi¬ 
nal  model  also  had  important  multiresolution  elements.  For  example, 
deployment  of  forces  could  be  governed  by  a  strategic-mobility  model 
or  by  a  simple  mechanism  in  which  the  analyst  essentially  scripted 
arrival  time  of  units.  Such  alternatives  also  existed  for  mobilization.  In 
addition,  it  was  possible  to  use  “type  units”  and  to  scale  their  capabili¬ 
ties  up  or  down  with  a  few  parameters,  rather  than  specifying  myriad 
named  units  and  their  many  characteristics  in  a  large  database.  It  was 
also  possible  to  scale  the  assumed  “fighting  capability”  (due  to  morale, 
leadership,  and  the  like)  with  a  few  parameters.21  As  the  model  was 
used  in  later  years,  however,  the  optional  simplicity  fell  into  disuse 
as  users  focused  on  getting  the  detailed  databases  “right”  (meaning 
agreed-upon)  for  running  standardized  case.22 

Analytic  organizations  with  large  investments  in  complicated 
models  should  consider  this  type  of  approach,  but  the  considerable 
overhead  involved  in  comprehending  the  workings  of  a  large  model 
and  maintaining  the  expertise  may  instead  suggest  spinning  off  smaller, 
simpler  models  as  discussed  above.23 
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The  Special  Issues  of  Social-Science  and  Other  "Soft" 
Considerations 

Models  Relating  to  Terrorism  and  Insurgency 

Most  defense  modeling  focuses  on  what  are  loosely  called  “physics 
problems”  in  which  results  depend  on,  say,  mass,  momentum,  accuracy, 
and  time.  In  recent  years,  DoD  has  been  sensitized  to  the  overwhelm¬ 
ing  importance  of  softer  factors  and  processes  such  as  those  studied  by 
social  scientists.  The  most  tangible  evidence  of  a  changed  view  came 
from  the  operational  commanders  with  the  new  Army-Marine  Field 
Manual  on  counterinsurgency.24  Today,  any  educated  reader  of  news¬ 
papers  knows  that  the  United  States  shifted  to  a  “population-centric” 
strategy  that  drastically  altered  tactics,  rules  of  engagement,  and  mea¬ 
sures  of  effectiveness  in  Iraq  and  Afghanistan. 

The  dominant  models  in  DoD’s  tool  kit  as  of  2003  were  unable 
to  deal  with  such  issues.  A  number  of  model  developments  proceeded, 
some  of  them  using  modern  methods  such  as  agent-based  modeling. 
However,  DoD  officials  came  to  realize  that  they  had  little  visibility 
into  what  these  models  were  doing,  skepticism  about  their  basis  in 
social  science,  and  doubts  about  whether  they  were  useful  for  analysis. 
The  officials  took  a  step  back  and  asked  for  research  about  what  the 
social-science  literature  told  us  that  should  be  represented  in  modeling 
and  analysis.  One  result  was  an  integrative  science  review;  it  started  as 
a  traditional  critical  review  of  the  literature  but  ended  up  using  quali¬ 
tative  models  called  factor  trees  to  relate  different  strands  of  scholarly 
work  and  put  the  findings  into  what  aspired  to  be  a  relatively  compre¬ 
hensive  structure.25  Factor  trees  are  diagrams  that  arrange  the  factors 
contributing  to  a  phenomenon  in  an  approximate  hierarchy  of  detail 
so  that  one  can  see  a  small  set  of  top-level  factors,  but  with  these  deter¬ 
mined  by  many  lower-level  factors,  sometimes  at  several  layers  of  detail. 
They  also  indicate  the  directionality  of  influence  and  when  certain  fac¬ 
tors  are  thought  to  be  individually  necessary  to  achieve  a  higher-level 
effect. 

Although  qualitative,  the  factor  trees  proved  quite  useful  for  sum¬ 
marizing  much  of  the  social-science  knowledge  in  a  structured  way 
understandable  even  to  large  audiences  with  diverse  backgrounds. 
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Such  depictions  are  definitely  simple  models,  not  just  cartoons.  As  with 
traditional  models,  they  frame  the  problem,  identify  key  factors,  and 
support  simple  stories  or  narratives  helpful  in  making  decisions,  such 
as  identifying  the  “lines  of  approach”  in  planning.  They  do  not,  how¬ 
ever,  purport  to  provide  quantitative  information  or  make  predictions. 
Factor  trees  have  been  used  in  a  number  of  subsequent  studies,  most 
of  them  classified.  One  openly  published  stream  of  work,  on  public 
support  for  terrorism,  describes  empirical  verification  through  use  of 
case  studies  and  includes  a  “primer”  on  the  use  of  factor  trees  as  an 
appendix.26 

Recently,  a  colleague  and  I  took  a  next  step  by  implementing  the 
factor  tree  from  the  public-support  study  in  a  computational  model, 
which  required  thinking  much  more  rigorously  about  relationships.27 
By  intent,  the  model  (called  PSOT)  is  designed  to  make  it  difficult  to 
use  as  a  black-box  answer  machine.  Instead,  the  model  generates  multi¬ 
dimensional  displays  indicating  what  combinations  of  factor  values  are 
likely  to  lead  to  a  good,  bad,  or  indifferent  result  with  respect  to  public 
support  of  terrorism.  Despite  the  uncertainties  built  into  the  model 
(into  the  model’s  algorithms,  not  just  input  data!),  the  model  incor¬ 
porates  more  of  the  available  social-science  knowledge  than  the  purely 
qualitative  factor  tree.  The  nature  of  the  computational  model  (full 
documentation  in  a  visual-programming  language,  Analytica®)  makes 
it  easier  for  scholars  to  review  and  debate  the  substantive  content  of  the 
model,  to  reproduce  it  in  their  own  analytic  laboratories,  and  to  con¬ 
struct  alternatives  if  need  be.  Such  countermodeling  has  proven  valu¬ 
able  in  the  long-established  realm  of  MIT-style  System  Dynamics,  as 
was  first  illustrated  during  a  period  of  high  controversy  in  the  1970s.28 

The  Peace  Operations  Support  Model  Computer-Aided  War  Game. 
A  second  example  of  how  social-science  issues  can  be  incorporated  in 
models  is  PSOM,  developed  in  the  United  Kingdom  and  used  both 
there  and  by  the  U.S.  Joint  Staff  to  support  operations  planning  in 
Afghanistan.  In  contrast  to  the  PSOT  model,  which  is  in  the  spirit  of 
capability  modeling  for  a  particular  mission  area  (in  this  case,  public 
support  for  terrorism  and  insurgency),  PSOM  is  a  campaign-level 
model  that  includes  political  and  economic  factors.29  An  important 
element,  in  my  view,  is  that  PSOM  was  designed  to  permit  human 
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players  to  discuss  and  make  higher-level  decisions  about,  e.g.,  resource 
allocation  and  maneuver.  Thus,  PSOM  provides  the  background  struc¬ 
ture  for  what  can  be  essentially  a  higher-level  war  game  with  senior 
participants.  My  understanding  is  that  this  kind  of  structured  gaming 
has  proven  to  be  quite  informative.  PSOM  has  also  enjoyed  consider¬ 
able  verification  and  validation.  Although  PSOM  is  hardly  the  kind  of 
relatively  simple  “capabilities  model”  emphasized  in  this  monograph, 
I  mention  it  because  it — like  other  campaign  models — can  be  quite 
valuable  as  part  of  an  analysis  campaign.  Like  other  campaign  models, 
it  may  seem  opaque  to  the  casual  observer  but  be  amenable  to  explana¬ 
tion  and  even  exploration  by  its  expert  users. 

Soft  Factors  in  Models  Generally 

Although  the  wars  in  Iraq  and  Afghanistan  have  dramatized  the  “soft” 
aspects  of  counterinsurgency  and  counterterrorism,  it  is  less  appreci¬ 
ated  that  soft  factors  are  important — even  crucial — in  modeling  and 
analysis  of  traditional  warfare  as  well.  This  was  explored  in  a  Mili¬ 
tary  Operations  Research  Society  symposium  decades  ago.30  The  sym¬ 
posium  included  papers  in  several  classes:  human  factors  in  decision 
issues,  human  performance  models,  human  performance  and  availabil¬ 
ity  in  combat  environments,  combat  as  a  data  source,  and  representing 
human  performance  in  combat  models  and  simulators. 

As  an  example  from  the  theater  level  of  modeling,  for  the  decades 
during  which  U.S.  defense  analysis  focused  largely  on  the  Soviet  Union, 
most  studies  ignored  such  soft  factors  as  morale,  leadership,  training, 
readiness,  and  even  the  effect  of  surprise  on  operational  effectiveness. 
This  seems  to  have  been  the  result  of  modelers  seeking  to  take  a  “scien¬ 
tific”  approach  that  emphasized  quantification,  “rigor,”  and  reproduc¬ 
ibility.  It  probably  reflected  also  the  view  that  World  War  II  had  ulti¬ 
mately  been  driven  by  the  numbers  and  the  belief  that  such  large-scale 
war  could  be  represented  by  mathematics  such  as  Lanchester  equations 
and  more  sophisticated  computer  models. 

In  any  case,  as  analysis  moves  forward,  analysts  should  be  dis¬ 
abused  of  the  notion  that  “good”  analysis  ignores  soft  factors.  As  I 
learned  decades  ago  in  a  revelation  when  reading  a  book  by  MIT’s  Jay 
Forrester,  to  ignore  a  soft  factor  is  analytically  equivalent  to  assuming 
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that  its  effect  is  zero.  What  kind  of  approximation  is  that  when  we 
know  from  history  that  such  factors  often  have  factor-of-two  effects, 
or  more?31 

Soft  factors  can  be  easily  included  in  models  intended  for  para¬ 
metric  capabilities  analysis.  As  merely  one  example,  in  characterizing 
the  effective  sizes  of  forces  in  an  assessment  of  the  military  balance  or 
making  an  estimate  of  likely  war  outcome,  the  effectiveness  of  low¬ 
readiness  ground-force  reserves  can  be  represented  by  their  nominal 
effectiveness  times  a  multiplier  varied  from,  say,  1/2  to  1.  However 
crude  that  may  be,  it  is  better  than  ignoring  the  issue  and  assuming 
full  effectiveness  of  ill-trained  soldiers.  More  generally,  the  values  of 
the  various  soft  factors  are  just  additional  dimensions  of  uncertainty. 
To  fail  to  address  them  in  a  misguided  effort  to  be  more  “rigorous”  is 
analytically  unconscionable.32 


Myths  and  Realities  About  Validation 

Whenever  one  discusses  models  in  DoD,  it  becomes  necessary  at  some 
point  to  deal  with  the  dreary  subject  of  verification,  validation,  and 
accreditation.  Fortunately,  extensive  material  is  available  on  recom¬ 
mended  practices  due  to  efforts  by  a  DoD  office.33  It  is  worthwhile, 
however,  for  me  to  confront  some  common  myths  that  are  sometimes 
raised  to  argue  against  simple  modeling,  efforts  to  represent  soft  or  oth¬ 
erwise  uncertain  variables,  use  of  human  gaming,  and  other  methods 
that  I  have  suggested. 

Model  Validation 

•  Model  “validation”  must  be  understood  in  terms  of  both  the 
model  and  the  input  assumptions.34  The  models  themselves  may 
be  unable  to  represent  some  phenomena,  and  can  thus  be  falsified 
in  that  respect,  but  if  they  can  represent  the  phenomena,  results 
are  typically  driven  by  uncertain  input  assumptions  rather  than 
model  details. 
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•  What  sometimes  passes  for  model  validation  is  instead  standard¬ 
ization  of  assumptions,  which  establishes  useful  base  cases  for 
comparisons  but  says  nothing  about  accuracy. 

•  Empirical  validation  as  in  physics  and  engineering  is  not  feasible 
for  most  higher-level  analysis — or  even  for  major  components  of 
the  models.  Controlled  experiments  are  out  of  the  question  and 
historical  results  provide  only  ambiguous  information.  As  a  result, 
validation  depends  on  indirect  means  such  as  expert  reviews  of 
structure  and  model  behaviors. 

•  For  exploratory  analysis,  it  is  feasible  to  review  model  structures 
(are  the  models  able  to  represent  the  issues  of  interest  and,  if  so, 
are  the  representations  adequate  even  if  parameter  values  are 
uncertain?)  and  to  review  the  uncertainty  ranges  being  used  for 
exploration.  Insights  from  exploration  may  then  be  valid  even  if 
precise  prediction  is  impossible. 

•  As  a  practical  matter,  the  quality  of  analysis  (and  of  the  under¬ 
lying  models  and  input  data)  depends  on  (1)  the  quality  of  the 
people  and  organizations  involved,  (2)  the  time  available,  and  (3) 
the  extent  to  which  the  analysis  and  results  can  be  debated  in  peer 
review.  Quality  depends  on  the  talents,  education,  and  experience 
of  people;  the  objectivity  and  practices  of  their  organizations;  and 
review  practices.  It  must  be  possible  for  others  to  reproduce  analy¬ 
sis  and  to  conduct  competing  analyses  with  different  models  or 
input  assumptions.  Debate  is  very  helpful.  As  the  expression  goes, 
“Sunlight  is  the  best  disinfectant.” 

•  Quality  of  models  and  analysis  cannot  be  legislated  or  ensured  by 
formalized  and  sometimes  bureaucratic  processes.  Further,  as  with 
quality  control  in  manufacturing,  problems  discovered  late  in  the 
process  are  difficult  and  expensive  to  correct:  Great  care  needs  to 
be  taken  in  the  underlying  design  and  foundational  work. 

•  For  higher-level  work,  the  more  detailed  models  are  often  not 
more  accurate  than  well-conceived  simpler  ones — especially  if 
only  the  simpler  models  represent  effects  of  factors  such  as  the 
sides’  command  and  control,  fighting  capabilities,  motivation, 
and  leadership.  Further,  regardless  of  their  quality,  if  used  only 
with  standardized  point-case  assumptions,  results  from  detailed 
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models  are  often  quite  unreliable — more  so  than  relying  on  para¬ 
metric  analysis  with  simpler  models  to  understand  the  range  of 
possible  results. 

I  would  be  remiss  not  to  acknowledge  controversies  and  organiza¬ 
tional  hurdles.  In  particular,  anyone  attempting  to  use  relatively  simple 
capability  models  and  parametric  analysis  should  not  be  surprised  to 
find  the  work  heavily  criticized  by  those  using  detailed  models  and 
standardized  data.  Further,  it  may  not  be  easy  to  gain  consensus  on 
the  validity  of  the  parameter  ranges  used  for  the  exploratory  analysis. 
Indeed,  an  admonition  to  managers  is: 

•  Investment  is  needed  in  activities  to  develop  well-vetted  value 
ranges  for  key  parameters  so  that  exploratory  capabilities  analysis 
can  be  well  informed  and  reviewed. 

Such  activities  may  require  considerable  work  because  so  much  of 
the  more  readily  available  information  is  very  detailed  (e.g.,  the  sortie 
rates  for  each  individual  air  force  squadron  and  how  that  varies  with 
the  availability  of  maintenance  facilities’  additional  crews).  Also,  many 
of  the  important  “parameters”  of  capability  models  do  not  exist  in  the 
more  detailed  models  except,  perhaps,  as  outputs. 

Validity  of  Human  Games,  Historical  Analysis 

•  The  purposes  of  human  gaming,  consultation  with  operators, 
drawing  on  historical  cases,  and  the  like  are  typically  not  “predic¬ 
tion”  and  are  typically  not  an  appropriate  subject  for  “validation.” 
The  purposes  include: 

-  Identifying  factors  and  relationships  affecting  the  phenomena 
of  interest,  including  factors  such  as  trust  across  people  and 
organizations  that  are  supposed  to  operate  effectively  together, 
culture-specific  perspectives  and  taboos,  and  ways  in  which 
misperceptions  and  misunderstandings  arise.  Many  such  fac¬ 
tors  and  relationships  may  not  even  be  represented  in  math¬ 
ematical  models. 
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-  Discovering  flaws  in  systems,  such  as  defensive  systems,  by 
turning  loose  the  creative  minds  of  people  who  are  good  at 
finding  such  flaws.  In  defense  work,  this  may  involve  “red- 
teaming.”  In  the  commercial  world,  it  may  involve  competition 
of  “hackers”  attempting  to  be  the  first  to  find  ways  to  crack 
new  software  (such  as  Apple’s  new  fingerprint  reader). 

As  noted  by  Thomas  Schelling  years  ago  in  regard  to  nuclear  war¬ 
gaming,  insights  are  so  valuable  that  we  should  be  more  than  willing  to 
tolerate  the  shortcomings  of  war-gaming  if  such  gaming  generates  such 
insights.  This  said,  the  validity  of  the  insights  must  then  be  assessed 
separately  rather  than  assumed  as  a  result  of  the  game  experience.35 

Similar  observations  apply  when  drawing  on  personal  testimo¬ 
nies  and  anecdotes,  or  even  when  interviewing  experienced  operators. 
It  is  not  as  if  such  data  can  be  assumed  generally  valid,  but  they  add 
to  the  body  of  knowledge,  may  highlight  important  insights,  and  may 
provide  more  accurate  estimates  than  those  stemming  from  idealized 
models.  As  I  write  this,  I  find  it  incongruous  that  using  such  informa¬ 
tion  needs  to  be  defended.  The  fact  is,  however,  that  analysts  are  often 
taught  to  focus  only  on  information  characterized  as  rigorous,  quanti¬ 
fiable,  reproducible,  and  objective.  This  often  translates  into  prejudice 
against  softer  information. 

Analysis  Validation 

As  nearly  everyone  has  concluded  after  looking  into  verification  and 
validation  issues,  what  is  most  important  is  to  ensure  the  quality  of  the 
specific  analysis  rather  than  attempt  some  broad  validation  of  all  the 
underlying  models,  data,  and  other  tools.36  Many  of  the  same  meth¬ 
ods  apply,  such  as  using  competitive  methods,  assuring  transparency, 
reproducing  results,  debating,  and  so  on. 

•  Validating  an  analysis  is  increasingly  feasible  to  the  extent  that  a 
campaign-analysis  approach  has  been  taken  that  draws  on  differ¬ 
ent  types  of  models,  tools,  and  expert  views,  and  if  the  validation 
effort  deals  effectively  with  uncertainty  and  disagreement. 
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•  The  “validity”  of  an  analysis  should  probably  be  judged  based  on 
whether  it  provides  decisionmakers  the  information  that  analysis 
can  reasonably  provide  and  does  so  comprehensibly  and  usefully. 

This  last  point  is  intended  to  reinforce  the  observation  made  at  the 
monograph’s  outset,  that  decisions  draw  on  analysis,  but  also  on  other 
considerations.  Sometimes,  analysis  can  provide  answers,  as  with 
Option  X  will  not  work,  but  many  other  times  the  choice  among 
options  will  depend,  and  should  depend,  on  the  subjective  judgments 
of  policymakers  who  are  informed  by  analysis  but  should  also  take  into 
account  matters  outside  what  analysis  can  provide. 


Endnotes 

1  Aspects  of  a  capability  model  may  be  fine-grained,  as  with  the  large  linear- 
program  models  that  have  been  used  in  OSD  (CAPE),  the  Joint  Staff,  and  U.S. 
Strategic  Command  for  40-50  years  to  characterize  capabilities  of  strategic-nuclear 
forces.  The  databases  are  large,  the  targeting  strategy  complicated,  and  the  algo¬ 
rithms  sophisticated.  However,  results  are  driven  by  a  modest  number  of  parameters 
such  as  the  accuracy  of  a  class  of  weapons,  the  scenario  (e.g.,  who  goes  first),  and 
crude  aspects  of  targeting  strategy. 

2  Understanding  the  essential  elements  of  missions  and  operations  has  long  been 
a  core  of  “strategies-to-tasks”  thinking  (Kent,  2002,  Kent  et  al.,  2008).  It  is  also 
reflected  in  success-tree,  fault-tree,  and  mission-system-analysis  methods  in  my  own 
work  (Davis,  1994a,  2002).  I  became  convinced  that  success  trees  are  a  stronger 
form  of  communication  than  equivalent  fault  trees  after  discussions  with  General 
Larry  Welch  (USAF,  retired)  in  the  late  1990s. 

3  Publications  exist  on  the  mathematics  of  Lanchester  equations  (Taylor,  1983), 
application  to  fleet  tactics  (Hughes,  2000),  understanding  the  mysteries  of  ground- 
combat  aggregation  and  disaggregation  (Davis,  1995),  and  the  potential  value  of 
intelligence,  surveillance,  and  reconnaissance  for  ground  combat  (Darilek  et  al., 
2001).  Constant-coefficient  Lanchester  equations  are  used  but  seldom  useful  in 
ground-combat  applications  because  forces  and  conditions  change  and  losing  com¬ 
manders  break  off  battle  rather  than  allowing  their  forces  to  be  annihilated.  Such 
effects  can  be  accounted  for  in  campaign  models.  The  equations  have  often  been 
used  in  attempts  to  fit  historical-empirical  data,  but — after  countless  papers — it  is 
now  clear  that  they  provide  a  poor  basis  for  doing  so  (Lucas  and  Turkes,  2003).  This 
is  hardly  surprising  since  they  have  a  poor  basis  in  underlying  theory  for  reasons  that 
include  not  distinguishing  between  distinctly  different  phases  of  combat,  misrepre¬ 
senting  the  likely  “statistical  mechanics”  of  dynamic  battle,  and  not  recognizing  the 
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inherent  heterogeneity  due  to  different  attrition  mechanisms.  Some  of  this  has  been 
discussed  in  the  literature  (Davis,  Blumenthal,  and  Gaver,  1996). 

I  Layering  models  are  used  extensively  in  ballistic-missile  defense  studies  (Kent, 
1963;  Wilkening,  1999;  Wilkening  et  al.,  1989)  and  also  in  homeland  defense  stud¬ 
ies,  which  included  discussion  of  subtleties  (Jackson  et  ah,  2012). 

5  I  learned  this  approximation  in  the  1970s  from  the  late  Christopher  Nolen  of 
the  Institute  for  Defense  Analyses,  one  of  the  most  respected  systems  analysts  of 
the  time.  The  exponent  of  1/3  assumes  processing  of  weak  signals  that  can  shorten 
detection  range  (Toomay  and  Hannen,  2004).  I  thank  colleague  David  Vaughan  for 
pointing  me  to  this  reference  to  refresh  my  memories. 

6  Numerous  halt  studies  were  done  in  the  late  1990s  and  early  2000s  (Davis  and 
Carrillo,  1997;  Ochmanek  et  ah,  1998;  Davis,  Bigelow,  and  McEver,  2001;  Davis, 
McEver,  and  Wilson,  2002;  McEver,  Davis,  and  Bigelow,  2000).  The  influential 
Ochmanek  et  ah  study  used  a  simple  spreadsheet  simulation. 

Davis,  Bigelow,  and  McEver  2001. 

8  Davis  et  ah,  2012. 

9  Davis  and  O’Mahony,  2013. 

10  See  Quinlivan,  2003,  or  the  earlier  Quinlivan,  1995.  The  caution  that  such  work 
suggested  about  how  many  “boots  on  the  ground”  might  be  needed  in  Iraq  was  dis¬ 
cussed  with  policymakers  before  the  Iraq  war  (Bremer,  2006). 

II  This  discussion  is  reminiscent  of  a  1950s  comment  about  systems  analysis  by 
Herman  Kahn.  He  noted  that  competence  and  honesty,  while  desirable,  were  not 
enough  (Kahn  and  Mann,  1957). 

12  Even  in  relatively  simple  or  mid-level  capability  models,  it  may,  for  example,  be 
necessary  to  represent  command  and  control  that  adjusts  the  concept  of  operations 
to  suit  the  context  of  a  given  model  run.  This  may  involve  local  optimization,  a  more 
ambitious  game-theoretic  algorithm,  heuristic  rules,  or  a  combination.  Such  aspects 
of  modeling  are  unfamiliar  to  those  using  large  and  complex  models  in  which  con¬ 
cepts  of  operations  and  tactics  are  embedded  in  data  provided  from  external  sources. 

13  Two  examples  may  suffice.  If  F(X)  is  bimodal,  but  fitted  to  a  straight  line,  the 
straight-line  approximation  might  be  good  “on  average”  but  much  too  low  or  much 
too  high  for  many  values  of  X.  If  F(X,  Y)  were  X*Y,  then  using  a  linear-fit  approxi¬ 
mation  would  mean  that  increasing  X  or  increasing  Y  would  have  the  same  effect 
when,  in  reality,  if  X  or  Y  is  0,  the  result  is  0  no  matter  how  large  the  other  variable 
is. 


14  Davis  and  Bigelow,  2002. 
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15  See  Bigelow  and  Davis,  2003,  for  the  motivated  metamodel  approach,  which  has 
been  used  in  numerous  studies.  It  remains  unusual,  although  its  philosophy  of  using 
theory-informed  statistical  analysis  to  test  theory  rather  than  to  induce  principles 
from  data  has  precedent  from  the  hard  sciences,  as  noted  by  the  famous  statistician 
George  Box  (Box,  1979,  2000). 

16  Davis,  Bigelow,  and  McEver,  2001. 

17  McEver,  Davis,  and  Bigelow,  2000;  Davis,  Bigelow,  and  McEver,  2001 

18  Davis,  Bankes,  and  Egner,  2007. 

19  The  strengths  and  weaknesses  of  theory-driven  and  data-driven  approaches  are 
discussed  in  Davis,  2009,  a  shortened  and  improved  version  of  similar  discussion  in 
Davis  and  Cragin,  2009. 

20  As  an  example,  one  can  gather  time-series  data  from  a  theater  of  combat  opera¬ 
tions  and,  without  thinking  about  the  operations,  strategy,  or  changes  in  either, 
look  for  correlations  with,  say,  the  number  of  friendly  forces.  But  why?  Why  would 
a  commander  or  other  consumer  of  analysis  permit  analysis  that  ignores  such  first- 
order  considerations? 

21  Wild,  Howe,  and  Davis,  1989.  For  a  fuller  description  of  the  original  Joint  Inte¬ 
grated  Contingency  Model  (then  referred  to  as  the  Main-Theater  Combat  model) 
see  Bennett  et  al.,  1988. 

22  This  is  discussed  in  a  white  paper  done  for  OSD  (Program  Analysis  and  Evalua¬ 
tion)  (Davis  and  Henninger,  2007). 

23  RAND’s  Excel-based  START  model,  developed  by  Barry  Wilson,  was  a  spi¬ 
noff  of  the  Joint  Integrated  Contingency  Model  incorporating  certain  particular 
algorithms. 

24  Petraeus  and  Amos,  2006. 

25  Davis  and  Cragin,  2009. 

26  Davis  and  O’Mahony,  2013. 

27  Davis  and  O’Mahony,  2013. 

28  Greenberger,  Crenson,  and  Crissey,  1976. 

24  Body  and  Marston,  2011,  is  part  of  a  special  issue  of  The  Journal  of  Defense 
Modeling  (http://dms.sagepub.eom/content/8/2.toc)  devoted  to  the  Peace  Support 
Operations  Model  with  chapters  by  Noel  Wilde,  Howard  Body  and  Colin  Marston, 
Nathan  Hanley  and  Helen  Gaffney,  Paul  Strong,  Gemma  Warren  and  Patric  Rose, 
Helen  Gaffney  and  Alastair  Vincent,  and  Oliver  Talbot  and  Noel  Wilde  on  model 
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philosophy,  modeling,  the  strategic-integration  process,  strategic  communication, 
stabilization,  and  security-sector  reform. 

30  Military  Operations  Research  Society,  1989. 

11  This  was  documented  by  the  late  historian  Trevor  N.  Dupuy.  See,  particularly, 
Dupuy,  1987,  a  polished  version  of  an  earlier  book  that  was  criticized  by  some  in  the 
operations  research  community  who  obsessed  on  his  overusing  data  (which  he  did) 
while  ignoring  the  power  of  his  contributions.  The  United  Kingdom  has  also  done 
considerable  historical  research  on  related  matters,  as  illustrated  in  a  book  by  David 
Rowland  on  the  role  of  stress  in  battle  (Rowland,  2006). 

12  The  original  Joint  Integrated  Contingency  Model  included  many  such  soft  fac¬ 
tors,  which  enabled  analysis  of  such  matters  as  the  plausibility  of  a  rapid  victory  by 
the  Warsaw  Pact  and  the  value  in  conventional  arms  control  of  constraints  on  the 
readiness  of  reserve  forces  (Davis,  1988b). 

33  Over  the  years,  the  office  has  been  named  the  Defense  Modeling  and  Simula¬ 
tion  Office,  the  Modeling  and  Simulation  Coordination  Office,  and  the  Model¬ 
ing  and  Simulation  Technology  Office.  The  weblink  for  “Verification,  Validation, 
and  Accreditation:  Recommended  Practices  Guide  (RPG)”  is  currently  http:// 
www.msco.mil/VVA_RPG.html.  Early  efforts  producing  this  material  were  led 
by  Simone  Youngblood  and  had  a  number  of  respected  contributors  (http://vva. 
msco.mil/Background/Credits.htm).  The  guide  does  not  deal  much  with  validating 
exploratory  analysis  as  emphasized  in  this  paper,  and  says  little  about  social-science 
modeling,  but  some  literature  on  that  issue  is  available  (McNamara,  Trucano,  and 
Gieseler,  2011;  Bigelow  and  Davis,  2003;  Bankes  and  Gillogly,  1994;  McNamara, 
2010). 

34  This  is  recognized  in  DoD’s  work,  which  defines  validation  as  “The  process  of 
determining  the  degree  to  which  a  model  and  its  associated  data  are  an  accurate  rep¬ 
resentation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model.” 

35  Schelling,  1987. 

36  In  DoD’s  parlance,  the  key  is  Accreditation,  defined  as  the  “official  certification 
that  a  model,  simulation,  or  federation  of  models  and  simulations  and  its  associ¬ 
ated  data  is  acceptable  for  use  for  a  specific  purpose.”  This  information  is  online, 
but  with  websites  that  change.  As  of  January  20,  2014,  see  http://www.msco.mil/ 
VVA_RPG.html. 
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Finding  Suitable  Options 


Challenges  in  Finding  Good  Options 

Previous  chapters  have  discussed  analysis  and  analysis  tools  but  another 
issue  is  finding  the  options  to  be  compared.  Frustrated  policymakers 
frequently  grumble  about  less-than-impressive  options — options  that 
perhaps  are  mere  extrapolations  of  what  their  organizations  have  been 
doing  in  the  past.  They  seek  options  that  solve  problems,  perhaps  with 
new  ways  of  thinking.  What  can  analysis  do?  Although  the  etymology 
of  “analysis”  may  suggest  otherwise,1  I  assume  in  this  monograph  that: 

•  Analysis  should  involve  not  just  decomposition,  but  also  integra¬ 
tion  and  synthesis  for  a  complex  world.  Such  analysis  requires 
creativity  and  a  sense  of  strategy. 

Analysts  may  help  find  good  options  in  many  ways.  During  a 
period  of  austerity,  for  example,  a  cost-sensitive  analyst  might  suggest 
efficiencies;  an  operator  or  a  policy  analyst  might  suggest  reducing 
objectives.  Flowever,  a  technologist  might  suggest  new  ways  of  doing 
things,  in  which  case  the  operator  might  suggest  a  new  concept  of 
operations  to  exploit  the  technology,  and  the  policy  analyst  might  then 
see  it  as  possible  to  maintain  objectives  but  accomplish  them  with  a 
new  strategy  based  on  the  new  concept  of  operations.  The  creative  ten¬ 
sion  of  working  across  stovepipes  of  policy,  operations,  technology,  and 
force-building  should  be  evident.  This,  indeed,  has  been  crucial  in  the 
past.  Table  5.1  sketches  some  of  the  relationships  over  time,  although 
the  reader  is  cautioned  to  remember  that  the  technologies  typically 
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Table  5.1 

Contributors  to  Strategic  Options:  A  Few  Examples  of  Synergy 


High  Strategy 

Concepts  of  Operation 

Programs 

Technologies 

Nuclear 

Assured-retaliation 
nuclear  theory 
(1960s-present) 

Nuclear  targeting  from  postures 
permitting  assured  response 

Triad  of  nuclear  forces 

Inertial  guidance  systems;  multi-reentry 
vehicles,  compact  nuclear  weapons; 
strategic  command  and  control 

Second-strike 

counterforce 

(1970s) 

Threaten  adversary's  nuclear  systems 
primarily  with  invulnerable  or  second- 
strike  systems 

Air-launched  cruise  missiles,  Trident  II 
missiles 

+  Even  better  inertial  guidance  systems 
and  terrain  contour  matching 

Countervailing 

strategy 

(1972-present) 

Deny  any  plausible  adversary 
leader  a  plausible  theory  of  victory; 
demonstrate  technological  superiority 

Mobile  intercontinental  ballistic  missiles 
(never  deployed);  stealth 

+  Earth-penetrating  warheads 
+  Stealth  technology 

Global  Military  Strategy 

Backstop  deterrence  in 
Europe  with  credible 
nuclear  forces 
(1950s-1970s) 

Initial  conventional  defense  In 

Europe's  Central  Region;  limited 
nuclear  options 

Limited  conventional  forces,  Pershing  II, 
ground-launched  cruise  missiles 

Advanced  guidance  systems 

Strengthen  conventional 

deterrence 

(1970s,  1980s) 

Rapid  reinforcement;  move  toward 
ability  to  interdict  Soviet  forces  via 
air-land  battle 

Prepositioning  of  Materiel  Configured 
to  Unit  Sets  (POMCUS)  program; 

Multiple  Rocket  Launching  System;  long- 
range  precision  fires  (Air  Force,  Army) 

Global  Positioning  System;  command, 
control,  computers,  surveillance,  and 
reconnaissance;  laser-guided  bombs; 
precision-guided  munitions 

Horizontal  escalation 
(1980s) 

Threaten  Soviet  bastions 

Nuclear  attack  submarines 

Reduced  acoustic  signatures 

Broaden  deterrence 
concept  in  East  Asia 
(201 3-?) 

Air-sea  battle 

Conventional  prompt  global  strike; 
new  long-range  bomber 

Miniature  weapons,  extreme  accuracy; 
intelligence,  surveillance,  and  recon¬ 
naissance;  stealth 

Counterterrorism  and  Counterinsurgency 

Population-centric 
counterinsurgency  + 
attack  of  leadership 

New  counterinsurgency  manual; 

Special  Operations  Command  direct 
action 

Predators,  Reapers,... 

Unmanned  aerial  vehicles  and  combat  air 
vehicles 

74  Analysis  to  Inform  Defense  Planning  Despite  Austerity 


Finding  Suitable  Options  75 


evolved  over  decades  and  were  not  necessarily  driven  by  the  anticipa¬ 
tion  of  strategic  changes  years  in  the  future. 


Recurring  Issues 

In  this  brief  discussion,  let  me  touch  on  three  recurring  issues  in  defin¬ 
ing  and  selecting  among  options:  (1)  how  much  capability  is  needed, 
(2)  how  the  attractiveness  of  options  depends  on  assumptions,  and  (3) 
what  concepts  of  operations  are  used  in  conceiving  and  evaluating  the 
options.  The  examples  are  primarily  at  the  strategic  rather  than  opera¬ 
tional  or  tactical  level. 

The  Perennial  Issue:  How  Much  Is  Enough? 

You  cannot  make  decisions  simply  by  asking  yourself  whether 
something  might  be  nice  to  have.  You  have  to  make  a  judgment 
on  how  much  is  enough. 


Robert  S.  McNamara,  April  20,  1963 

The  question  of  “How  much  is  enough?”  has  a  different  charac¬ 
ter  in  each  capability  area.  For  strategic  planning,  however,  it  often 
amounts  to  assessing  what  is  needed  to  favorably  affect  the  behavior  of 
some  potential  adversary  or  competitor.  Frequently,  this  is  discussed  in 
the  language  of  deterrence.  Suppose  that  the  United  States  has  some 
concern  about  a  potential  adversary  in  a  particular  region.  In  deciding 
how  much  capability  is  needed,  the  United  States  might  consider  at 
least  the  following  strategies: 

1.  Presence :  Maintain  enough  military  presence  to  show  that  the 
United  States  has  interests  in  the  region — even  though  it  is  not 
postured  for  immediate  defense. 

2.  Tripwire :  Maintain  and  position  forces  sufficient  to  assure  that 
aggression  would  very  likely  mean  military  confrontation  with 
U.S.  forces. 
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3.  Initial  defense :  Maintain  enough  military  capability  to  permit 
significant  defense,  at  least  temporarily.  A  purpose  here  would 
be  to  preclude  quick  and  simple  aggression,  something  that  may 
be  quite  valuable  because  conventional  aggression  has  often  been 
undertaken  with  the  expectation  of  quick  and  easy  victory.2 

4.  Substantial  defense :  Maintain  enough  military  capability  to 
permit  potentially  effective  and  successful  defense — enough  so 
that  the  adversary  would  have  serious  doubts  about  his  ability  to 
prevail  even  if  his  “best  estimate”  was  favorable.  This  capability 
might  be  sized,  for  example,  for  a  first  campaign  lasting  weeks.3 

5.  Robust  defense:  Maintain  enough  military  capability  to  deci¬ 
sively  defeat  aggression  with  confidence.  This  might  be  called 
“Deter  by  Denial.” 

Figure  5.1  indicates  a  fairly  generic  how-much-is-enough?  curve 
of  diminishing  returns  for  such  problems.  To  illustrate  how  the  figure 
is  to  be  read,  suppose  that  one  wants  to  evaluate  deterrence  skeptically, 
in  the  belief  that  robust  war-fighting  capability  is  ultimately  important 
(bottom  curve).  With  that  perspective,  there  is  surely  some  deterrence 

Figure  5.1 

How  Much  Is  Enough? 
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from  having  modest  capability,  but  to  have  a  high  level  of  deterrence 
requires  much  more  capability.  In  contrast,  someone  who  believes 
that  deterrence  is  existential  may  believe  that  the  quality  of  deterrence 
improves  quickly  with  capability  and  then  reaches  the  point  of  dimin¬ 
ishing  returns. 

It  is  often  important  to  have  options  suited  for  each  of  the  sev¬ 
eral  judgments  about  what  is  needed  for  deterrence.  Policymakers  must 
decide  the  nature  of  the  deterrence  being  attempted  and  will  draw  on 
nonmilitary  considerations  such  as  past  relations  with  the  country  in 
question,  plausible  incentives  or  disincentives  for  war,  and  the  degree 
to  which  deterrence  is  driven  by  economics,  international  relations,  or 
some  other  nonmilitary  factors. 

Robust  defense  based  on  war-fighting  capability  might  seem  to 
be  the  obvious  preference,  but  countervailing  considerations  some¬ 
times  exist.  First,  improving  one’s  “defensive”  capabilities  typically  is 
seen  by  the  adversary  as  posing  an  increasingly  worrisome  threat.  This 
is  the  famous  security  dilemma  of  political  science.4  Second,  the  cost 
of  robust  defense  can  be  extremely  high  and  the  need  is  questionable 
if  conflict  seems  sufficiently  unlikely,  especially  if  it  were  possible  to 
scale  up  the  defense  in  the  future  when  necessary.  Finally,  it  can  be 
argued  that  preparing  for  war  (no  matter  how  much  emphasis  is  given 
to  the  purpose  of  pure  defense)  may  make  war  more  likely  because  it  is 
attended  by  the  hardening  of  attitudes,  suspicions,  hatreds,  and  para¬ 
noia.  This  is  a  potential  concern  as  the  United  States  evolves  its  strategy 
with  respect  to  China,  which  is  a  competitor  but  one  that  may  never 
become  an  adversary.5  A  related  perennial  issue  is  what  fraction  of  the 
necessary  military  capability  can  realistically  be  provided  by  regional 
allies. 

The  next  layer  of  sophistication  in  considering  options  is  to  take  a 
broader  strategic  view.  It  may  be,  for  example,  that  the  adversary  would 
“really”  be  deterred  not  by  the  local  military  balance,  but  by  broader 
factors.  War  might  have  disastrous  economic  or  political  consequences. 
If  so,  then  less  military  capability  might  well  be  enough.  Most  impor¬ 
tant,  perhaps,  is  “existential  deterrence.”  If  an  adversary  believed  that 
conflict  would  escalate  to  general  war,  even  general  nuclear  war,  then 
he  would  be  deterred  even  if  he  were  confident  of  his  ability  to  win  an 
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initial  conflict.  Unfortunately,  it  is  not  evident  that  all  plausible  adver¬ 
sary  leaders  would  have  this  existential  view.  In  particular,  he  might 
not  deem  the  threat  of  nuclear  escalation  to  be  credible — the  primary 
challenge  in  attempting  to  extend  conventional  deterrence  by  threaten¬ 
ing  use  of  nuclear  forces. 

Dependence  on  Assumptions 

Even  if  we  have  decided  functionally  on  the  necessary  level  of  military 
capability,  which  option  will  appear  superior  may  depend  sensitively  on 
a  number  of  assumptions.  Consider  some  examples: 

•  Would  conflict  be  preceded  by  strategic  warning  and  mobilization? 

•  How  much  capability  and  determination  would  be  provided  by 
regional  partners? 

To  illustrate  how  consequential  these  are,  the  United  States  pos¬ 
tured  its  nuclear  forces  in  the  Cold  War  to  assure  the  ability  to  mas¬ 
sively  retaliate  against  the  Soviet  Union  even  if  the  Soviets  attacked 
first  without  warning  (the  “bolt  from  the  blue”).  The  Soviet  Union,  in 
contrast,  planned  with  the  expectation  that  any  conflict  would  come 
out  of  a  crisis  that  would  permit  alerting  forces.  Historically,  China 
has  also  not  exhibited  much  concern  about  a  bolt-from-the-blue  attack. 
Was  the  United  States  being  overly  conservative  and  economically 
wasteful?  Or  was  it  being  wise? 

As  a  second  example,  consider  how  apparent  force  requirements 
for  counterinsurgency  operations  have  changed  as  the  United  States  has 
decided  to  plan  for  the  bulk  of  effort  to  be  done  by  regional  partners. 

The  point,  of  course,  is  not  that  one  or  another  strategy  is  right, 
but  that  the  notion  of  capability  “requirements”  is  intimately  associated 
with  assumptions  and  strategies.  Thus,  option  development  should  rec¬ 
ognize  the  different  ways  that  policymakers  may  choose  to  proceed. 

Dependence  on  Concepts  of  Operations 

Another  pivotal  factor  in  answering  the  how-much-is-enough  question 
is  the  concept  of  operations  assumed  in  evaluating  an  option’s  capabili¬ 
ties.  This  is  not  a  theoretical  subtlety  enjoyed  by  ivory-tower  scholars, 
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but  a  core  issue  in  practical  planning — especially  in  periods  of  auster¬ 
ity  when  it  becomes  essential  to  be  creative  lest  our  capabilities  be  sav¬ 
aged  by  budget  cuts. 

Some  principles  apply  here  as  well,  principles  based  on  historical 
experience: 

•  When  new  technologies  are  introduced,  existing  organizations 
(and  budgeteers)  often  evaluate  them  by  estimating  how  much 
more  efficiently  the  technologies  would  allow  the  organizations 
to  operate  using  current  organization  and  practices.  The  result  is 
often  to  underestimate  the  significance  of  the  new  capabilities.6 

•  If  new  concepts  of  operations  and  new  organizations  are  created 
to  exploit  the  new  technologies,  it  is  frequently  possible  to  do  much 
more  with  much  less — i.e.,  to  be  far  more  effective  (in  at  least 
some  dimensions)  while  using  many  fewer  people  and  machines. 

•  In  periods  of  economic  cutback,  failure  to  introduce  such  innova¬ 
tions  may  have  seriously  disproportionate  effects  on  operational 
capabilities.  As  mentioned  above,  a  10  percent  cutback  may  mean 
a  20  percent  cutback  in  the  number  of  employable  force  units. 

To  be  less  abstract,  consider  some  examples: 

•  Precision  weapons  have  drastically  reduced  the  number  of  aircraft 
needed  for  devastating  attacks,  whether  by  fighters  or  bombers. 

•  High-technology  networking  and  command  and  control,  coupled 
with  new  special-operations  tactics,  have  had  a  major  impact  on 
counterinsurgency  operations  in  Afghanistan,  where  U.S.  ground 
forces  have  been  far  fewer  in  number  than  what  has  been  tradi¬ 
tionally  needed  for  these  type  of  operations. 

•  U.S.  Army  operations  are  now  “brigade  centric”  rather  than  divi¬ 
sion  centric 

•  The  future  of  ground-force  operations  is  likely  to  be  increasingly 
special-operations  centric. 

•  The  United  Kingdom’s  army,  which  has  been  suffering  severely 
from  budget  cutbacks,  is  restructuring  to  change  fundamentally 
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how  it  uses  reserves  and  what  assumptions  it  makes  about  the 

kinds  of  operations  in  which  the  its  army  will  participate.7 

In  our  day-to-day  world,  there  are  numerous  examples  of  “revolu¬ 
tions,”  such  as  the  ones  we  associate  with  Federal  Express,  Walmart, 
and  Amazon.  A  common  feature  of  these  and  the  military  items  above 
is  that  technology,  concepts  of  operations,  and  organizations  all  had  to 
change. 

The  reason  for  discussing  these  matters  is  to  highlight  the  neces¬ 
sity  in  analysis  of  going  beyond  current  ways  of  doing  business. 

We  can  confidently  assume  here  that  many  good  ideas  exist  for 
discrete  matters,  such  as  modernizing  an  aircraft’s  avionics,  reduc¬ 
ing  the  rate  of  growth  of  personnel  benefits,  and  improving  logisti¬ 
cal  efficiencies.  What  follows  is  about  how  these  “building  blocks” 
might  be  put  together.  For  higher-level  decisionmaking,  after  all,  the 
options  to  be  considered  are  normally  composites,  as  with  a  proposed 
defense  program  with  innumerable  facets  but  an  overall  “tilt”  on  the 
margin  toward  a  particular  strategic  direction.8  Sometimes,  top  leaders 
must  decide  on  something  more  narrow,  such  as  which  capabilities  to 
acquire  for  a  particular  mission  or  whether  to  proceed  with  a  particu¬ 
larly  expensive  system.  Even  there,  however,  they  deal  with  composite 
options.  For  example,  going  ahead  with  a  particular  program  such  as 
a  space-based  surveillance  system  implies  going  ahead  with  numerous 
component  programs  involving,  e.g.,  the  platform,  sensors,  launchers, 
and  support  structure. 

Flow  composite  options  come  to  be  created  in  DoD  is  more  of 
an  art  than  a  science.  It  often  depends  on  the  knowledge,  context- 
sensitive  intuition,  and  skill  of  the  staff  in  OSD  (Cost  Assessment  and 
Program  Evaluation),  the  services’  analytic  organizations,  and  the  Joint 
Staff’s  J-8.  The  composite  options  are  often  rather  crudely  defined.  One 
option,  for  example,  might  call  for  a  relative  tilt  toward  the  Asia-Pacific 
region  and  away  from  manpower-intensive  counterinsurgency  and,  at 
the  same  time,  prioritizing  acquisition  programs  accordingly.  Many 
other  features  might  be  part  of  that  option,  but  would  not  be  explicit 
in  the  title  or  short-form  description  of  the  option. 
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Special  Implications  for  a  Period  of  Austerity 

The  admonitions  of  this  section  are  especially  important  in  periods  of 
austerity  and  competition  among  services.  In  such  periods,  organiza¬ 
tions  often  try  to  hold  onto  as  much  of  their  current  status  as  possible, 
limiting  cuts  in  personnel  and  equipment  and  planning  to  continue 
with  the  same  approaches  as  in  the  past  to  the  maximum  extent  possi¬ 
ble.  When  budget  cuts  are  significant,  they  may  call  for  reducing  their 
responsibilities — i.e.,  eliminating  some  missions  or  lowering  expecta¬ 
tions  of  performance.  In  defense  planning  as  in  other  domains,  it  is 
sometimes  possible  instead  to  go  about  missions  differently  with  an 
infusion  of  technology,  new  concepts  of  operation,  and  new  organiza¬ 
tional  approaches.  In  some  cases,  it  is  even  possible  to  do  more  with 
less.  Where  this  is  possible,  however,  there  must  be  new  investments 
and  the  ability  to  pay  for  the  processes  of  change.  That,  in  turn,  typi¬ 
cally  implies  making  even  larger  cuts  in  personnel  expenses  and  cur¬ 
rent  equipment. 

Without  prejudice  as  to  whether  such  options  make  sense  at  the 
present  time,  examples  in  the  current  defense  world  include  (1)  further 
reducing  the  size  of  ground-force  building  blocks  to  something  more 
battalion-like  in  size  (or  even  smaller)  and  efficient  use  of  new  tech¬ 
nology,9  (2)  further  reducing  the  size  of  air  forces’  building  blocks  to 
reflect  the  greater  capability  of  new  systems  and  the  absence  of  formi¬ 
dable  air-force  threats,  (3)  moving  more  drastically  toward  crew-rota¬ 
tion  approaches,  and,  as  in  the  United  Kingdom,  and  (4)  reconceiving 
the  use  of  reserve-component  forces,  as  in  more  fully  integrating  them 
rather  than  seeing  them  as  a  strategic  reserve.10  All  of  these  have  been 
suggested;  all  have  evident  or  potential  shortcomings,  as  well  as  vir¬ 
tues.  The  point  here  is  merely  that — in  times  such  as  the  present — the 
options  under  consideration  should  go  well  beyond  scaling  down  cur¬ 
rent  structure. 


Building-Block  and  Composite  Options 

In  discussing  option  development  for  capability  options,  level  of  analysis 
matters.  The  U.S.  Joint  Staff  has  carefully  built  an  elaborate  taxonomy 
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of  joint  capability  areas  with  multiple  layers  of  detail.11  Within  each 
capability  area,  at  whatever  level  of  detail,  there  is  a  need  to  understand 
the  challenges,  identify  the  capabilities  that  may  be  needed,  develop 
options,  and  choose  among  options.  For  relatively  detailed  joint  capa¬ 
bility  areas,  such  as  offensive  subsurface  warfare,  the  options  will  also 
be  relatively  detailed — perhaps  well  below  the  interest  level  of  the  Sec¬ 
retary  of  Defense  or  Chairman  of  the  Joint  Chiefs  of  Staff. 

An  interesting  role  for  analysts  arises  because,  when  decisions  are 
made  at  a  given  level,  the  options  considered  are  usually  composites  of 
the  lower-level  building-block  options.  For  example,  in  making  deci¬ 
sions  about  a  program  for  undersea  warfare,  the  options  would  proba¬ 
bly  all  be  composites  of  building-block  options  for  antisubmarine  war¬ 
fare,  subsurface  warfare,  and  mine  warfare.  These  composite  options 
would  differ  in  the  relative  resources  allocated  to  each,  and  perhaps  in 
the  character  of  the  solutions.  If  there  was  a  uniquely  best  building- 
block  option  for  each  of  the  lower-level  categories,  then  the  options 
would  differ  only  in  the  relative  resource  allocation.  In  reality,  however, 
there  may  be  ways  to  achieve  joint  capabilities  that  are  not  just  a  matter 
of  assembling  the  building  blocks  offered  up  by  the  services. 

For  each  capability  area,  regardless  of  detail,  certain  information 
is  needed.  The  mission  has  to  be  understood  and  characterized  analyti¬ 
cally.  The  effectiveness  of  an  option  for  accomplishing  that  mission  (as 
measured  perhaps  by  a  number  of  separate  metrics)  needs  to  be  under¬ 
stood  as  a  function  of  the  investment  made  in  the  relevant  program(s). 
In  some  cases,  “requirements”  will  be  relatively  fixed  and  objective,  but 
in  other  cases  requirements  will  be  decided  based  on  looking  at  what 
can  be  accomplished  as  a  function  of  investment  and  deciding  how 
much  is  enough. 

Although  the  joint  system  for  capabilities  assessment  and  develop¬ 
ment  is  highly  organized,12  no  rule  book  can  specify  how  to  conceive 
options,  much  less  decide  what  is  “best.”  Some  recurring  themes  and 
principles,  however,  bear  mention.  Because  the  United  States  is  in  a 
period  of  austerity  for  defense  planning,  it  is  essential  to  make  choices. 
One  criticism  of  capabilities-based  planning  has  been  that  too  often 
its  practitioners  have  been  eager  to  identify  shortfalls  and  ask  for  more 
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money  without  adequately  making  tough  choices.  During  the  2000s, 
it  sometimes  was  referred  to  as  a  blank-check  approach. 


A  New  Method  for  Developing  Options 

The  one  contribution  to  option  development  that  I  intend  for  this 
monograph  is  to  suggest  a  new  method  for  developing  composite 
options.13  In  a  sense,  the  method  seeks  to  formalize  analysis  for  what 
has  long  been  done  ad  hoc  by  creative  staff.  The  methodology  can  be 
applied  literally  for  some  problems,  whether  for  attention  at  the  Secre¬ 
tary  of  Defense  level  in  choosing  strategy  or  at  the  Under  Secretary  of 
Defense  (Acquisition,  Technology,  and  Logistics)/Vice  Chairman  level 
in  deciding  on  how  to  proceed  in  achieving  a  desired  mission  capa¬ 
bility.  In  other  cases  (probably  most  cases),  the  concepts  behind  the 
approach  represent  a  paradigm  that  may  be  useful  even  if  the  analytic 
machinery  is  not  used. 

The  underlying  analytic  concepts  are  these,  as  sketched  in  Figure 

5.2:14 

1.  Consider  initially  a  vast  range  of  possible  composite  options 
(i.e.,  the  many  possibilities  inherent  in  a  Chinese  menu),  thereby 
going  beyond  conventional  wisdom  and  prejudice  as  to  what 
might  be  considered. 

2.  Filter  out  the  vast  bulk  of  such  composite  options  because  they 
are  distinctly  inconsistent  with  any  of  the  strategies  under  con¬ 
sideration  with  respect  to  multicriteria  effectiveness,  cost,  or 
both. 

3.  Identify  the  small  subset  of  composite  options  that  are  poten¬ 
tially  the  best  in  cost-effectiveness  for  the  various  cost  levels 
under  consideration  when  uncertainties  are  accounted  for. 

4.  Plan  to  examine  these  surviving  options  in  much  more  detail  as 
discussed  in  Chapter  Six. 

The  new  features  are  (1)  using  computer  methods  to  generate 
more  possibilities,  (2)  filtering  by  multicriteria  effectiveness  measures 
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Figure  5.2 

A  New  Way  to  Construct  Policy-Level  Composite  Options 
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suggested  by  the  range  of  strategies  still  being  considered,  and  (3)  rec¬ 
ognizing  that  uncertainties  on  which  there  are  disagreements  affect 
estimates  of  multicriteria  effectivenesses  and  cost. 

The  classic  approach  is  as  suggested  in  the  left  pane  of  Figure  5.3. 
After  plotting  points  against  their  effectiveness  (vertical  axis)  and  cost 
(horizontal  axis),  one  draws  the  Pareto-Optimum  curve,  often  called 
the  “efficient  frontier.”  All  options  on  that  frontier  are  optimal:  For 
any  given  cost  level,  the  options  on  the  frontier  have  highest  effective¬ 
ness.  And,  at  any  given  effectiveness  level,  the  options  on  the  frontier 
are  least  costly.15  This  classic  approach  has  many  practical  shortcom¬ 
ings.  Estimates  of  both  effectiveness  and  cost  are  likely  to  be  squishy 
and  even  relative  estimates  may  be  significantly  wrong.  As  shown  in 
the  right  pane,  suppose  that  the  cost  of  Option  A  turns  out  to  be  a  bit 
higher.  That  means  that  the  previously  inferior  Option  E  is  now  on  the 
efficient  frontier  and  Option  A  should  be  discarded.  Further,  the  fron¬ 
tier  curve  itself  shifts.  The  right  pane  also  shows  what  happens  if  one 
decides  that  Option  G  is  so  close  to  Option  C  that  we  do  not  want  to 
discard  it. 

The  first  generalization  is  to  keep  options  that  are  “close”  to  the 
efficient  frontier,  as  suggested  in  the  bottom  pane  of  Figure  5.3.  This 
means  that  the  frontier  becomes  a  fuzzy  region  with  some  thickness 
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Figure  5.3 

Generalized  Approach  for  Finding  Options  on  the  Efficient  Frontier 

b)  If  A  costs  more  and  G  is 
a)  Initial  approach  considered  "close  enough" 

Efficient  frontier  Efficient  frontier 


c)  Generalized  concept 
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(perhaps  10  percent  or  so  along  both  the  effectiveness  and  cost  dimen¬ 
sions,  although  the  drawing  is  just  notional).  This  has  considerable 
effects  on  analysis.  A  further  generalization,  given  in  Chapter  Six,  has 
an  even  more  dramatic  effect. 

A  first  tangible  illustration  of  this  method  was  in  a  study  proto¬ 
typing  the  approach  for  two  Under  Secretaries  of  Defense  (Acquisition, 
Technology,  and  Logistics)  (Michael  Wynne  and  his  successor  Ken- 
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neth  Krieg).  The  prototype  problem  was  prompt  conventional  global 
strike  for  which  a  variety  of  different  mission  acquisition  strategies  were 
being  considered.16  That  example  will  be  discussed  further  in  Chapter 
Six. 

With  these  suggestions  on  option  development,  let  us  now  turn  to 
the  important  subject  of  portfolio  analysis. 


Endnotes 

1  In  scholarly  work,  “analysis”  typically  is  defined  as  “resolution  of  anything  com¬ 
plex  into  simple  elements,”  and  is  contrasted  with  synthesis.  This  meaning  traces 
back  to  Medieval  Latin  and  Greek.  Merriam-Webster,  however,  recognizes  as  its 
fourth  definition:  “an  examination  of  a  complex,  its  elements,  and  their  relations,” 
which  is  more  consistent  with  the  broad  meaning  in  policy  analysis.  I  discuss  this 
because  critics  of  “analysts,”  particularly  of  system  analysts  and  program  and  budget 
analysts,  often  accuse  analysts — sometimes  with  justification — of  being  hopelessly 
reductionist  with  no  sense  of  strategy. 

2  Mearsheimer,  1983. 

3  With  regard  to  NATO’s  Central  Region,  an  often-discussed  goal  was  being  able 
to  defeat  the  Soviet  first  and  second  strategic  echelons  over  the  course  of  a  month  or 
so  of  combat.  For  strategic-nuclear  planning,  proponents  of  the  Strategic  Defense 
Initiative  during  President  Reagan’s  tenure  often  argued  that  such  a  defense,  while 
imperfect,  would  greatly  raise  Soviet  uncertainties  about  the  ability  to  prevail. 
Others  argued  that  the  Soviet  uncertainties  were  already  extremely  high  and  that 
an  imperfect  strategic  defense  initiative  would  accomplish  little  if  anything  except 
support  the  economy.  The  same  people,  however,  often  supported  having  substan¬ 
tial  survivable  counterforce  capability  because  that,  in  their  view,  did  indeed  raise 
uncertainties. 

4  Jervis,  1976. 

5  See  the  discussion  in  two  recent  studies  led  by  David  Gompert,  written  soon  after 
he  served  as  Acting  Director  of  National  Intelligence  (Gompert,  2013;  Gompert  and 
Saunders,  2011). 

6  Early  computers  were  evaluated  for  cost-effectiveness  relative  to  typewriters  and 
hand  calculators  with  no  recognition  that  work  practices  would  change  drastically. 
When  aircraft  carriers  first  entered  the  fleet,  they  were  seen  as  able  to  improve  scout¬ 
ing,  with  battleships  continuing  to  be  the  core  capability. 


7  Carr-Smith,  2013. 
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8  Informing  portfolio  balance  with  strategy  is  illustrated  in  Johnson  et  al.,  2012, 
and  Davis  et  al.,  2008a. 

9  Arquilla,  2008. 

10  Carr-Smith,  2013. 

11  See  http://www.dtic.mil/futurejointwarfare/jca.htm  for  information  on  joint 
capability  assessment,  including  the  taxonomy. 

12  Joint  Chiefs  of  Staff,  2011. 

13  See  Davis  et  al.,  2008b;  Davis,  Shaver,  and  Beck,  2008,  for  the  analytic  method 
and  an  application. 

14  The  ideas  were  developed  collaboratively  with  colleague  Russell  Shaver. 

15  Markowitz,  1952;  Markowitz,  Sharpe,  and  Miller,  1991. 

16  See  Davis  et  al.,  2008b;  Davis,  Shaver,  and  Beck,  2008;  and  related  discussion 
from  a  larger  workshop  held  by  OSD  (Acquisition,  Technology,  and  Togistics) 
(Porter,  Bracken,  and  Kneece,  2007). 
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Portfolio  Analysis  In  Comparison  with  Other  Approaches 

Even  if  one  has  a  strong  base  of  underlying  analysis,  the  issue  remains 
of  how  to  evaluate  the  options  and  choose  among  them.  Historically,  a 
variety  of  methods  have  been  used  to  compare  options.  These  include 
equal-effectiveness  comparisons  (often  with  optimization),  equal-cost 
comparisons,  and  mixed  comparisons.  The  kind  of  portfolio  analysis 
described  here  falls  into  the  third  class.  It  is  especially  suitable  for  stra¬ 
tegic  and  other  high-level  decisions  in  which  decisionmakers  have  to 
deal  with  multiple  objectives,  uncertainties  and  disagreements  about 
“everything” — including  the  criteria,  alleged  “requirements”  for  each, 
costs,  and  risks.  Often,  the  solution  “emerges”  as  the  result  of  process, 
rather  than  being  the  solution  to  a  well-posed  mathematics  problem. 
Optimization  methods  can  be  highly  valuable  along  the  way,1  but  the 
most  important  matters  are  decided  more  subjectively,  especially  at  the 
top  levels. 


Basic  Concepts 

The  approach  to  portfolio  analysis  described  here  is  well  developed 
and  documentation  exists.2  It  is  motivated  by  recognition  that  strate¬ 
gic  decisionmakers  have  multiple  responsibilities  and  multiple  classes 
of  capability  to  consider.  Their  challenge,  at  budget  time,  is  to  “bal¬ 
ance”  their  program  so  that  they  are  attending  adequately  to  all  of  their 
responsibilities  while  giving  special  emphasis  in  one  way  or  another  to 
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some.  This  requires  a  form  of  multicriterion  analysis.  Table  6.1  sum¬ 
marizes  ways  in  which  this  seemingly  straightforward  way  of  look¬ 
ing  at  things  differs  from  narrowly  construed  decision  analysis  in 
which  one  constructs  a  common  scale  of  utility  across  objectives,  uses 
linear  weighting  but  with  constraints  to  keep  results  within  acceptable 
bounds,  emphasizes  “hard”  or  “objective”  evaluations,  and  optimizes 
cost-effectiveness  for  a  base  case,  perhaps  doing  some  sensitivity  testing 
(e.g.,  varying  weights).  In  portfolio  analysis,  the  style  is  to  recognize 
multiple  incommensurate  objectives  without  attempting  to  combine 
them;  to  be  on  the  watch  for  nonlinear  relationships  such  as  in  systems 
with  multiple  critical  components;  to  use  both  objective  and  subjec¬ 
tive  evaluations  with  unabashed  judgments  playing  a  big  role;  to  seek 
merely  to  “balance”  rather  than  optimize  (the  meaning  of  which  is 
doubtful  here),  with  the  intent  being  to  address  all  of  the  multiple  cri¬ 
teria  “adequately”;  and  to  confront  uncertainty  from  the  outset. 


Table  6.1 

What  Makes  Portfolio  Analysis  Different 


Simple  Decision  Analysis 

Portfolio  Analysis 

Objectives 

Single,  overarching 
aggregate  objective 

Multiple,  incommensurate 
objectives 

Relative  emphasis 
of  objectives 

Linear  weights;  constraints 

Sometimes,  complex  nonlinear 
relationships  to  be  discovered 
rather  than  understood  a  priori 

Evaluations  and 
emphasis 

"Objective"  evaluations  and, 
often,  a  sense  of  once-and- 
for-all  evaluation 

A  mix  of  objective  and  subjective 
evaluations  with  judgments  and 
relationships  evolving  over  time 

Governing  ideal 

Optimizing  for  cost- 
effectiveness,  often 
minimizing  cost  for  equal- 
effectiveness  options 

Balancing  to  achieve  good- 
enough  effectiveness  by  all 
criteria,  while  remaining  within  a 
top-line  budget  but  reconsidering 
component  budgets 

Treatment  of 
uncertainty 

Base  case  treated  as 
authoritative  design  point; 
limited  sensitivity  analysis; 
reporting  of  assumptions 

Exploratory  analysis  over  all 
major  dimensions  of  uncertainty; 
encourage,  aid-hedging  for 
FARness 
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A  Simple  Illustration 

Table  6.2  illustrates  the  concept  with  a  simplified  example  drawn  from 
a  study  on  prompt  conventional  global  strike — a  study  that  had  high- 
level  DoD  interest  because  of  its  strategic  consequences.  Although  the 
values  are  contrived,  the  display  claims  that  the  baseline  capability  for 
prompt  conventional  global  strike  is  poor  (red),  that  it  can  be  improved 
markedly  for  attacking  terrorist  leaders  in  a  short-duration  meeting 
about  which  the  United  States  gains  intelligence,  but  is  subject  to  the 
terrorists  having  shorter  meetings  or  even  greater  secrecy  about  where 
they  are  held.  The  bomber  option  has  some  capability  to  attack  weap¬ 
ons  of  mass  destruction  facilities  given  adequate  intelligence,  but  not 
without  that  intelligence.  Sensors,  by  themselves,  do  nothing,  but  in 
combination  with  the  missile  or  bomber  option  provide  some  ability 
to  attack  mobile  intercontinental  ballistic  missiles,  although  certain 
countermeasures  would  eliminate  that  capability. 

Table  6.2  could  be  the  basis  for  a  substantive  discussion  of  the 
options’  nominal  cost  and  effectiveness.  Such  stoplight  charts,  how¬ 
ever,  are  notorious  for  often  having  no  underlying  basis.  An  important 
feature  of  the  portfolio  analysis  approach  is  that  it  should  be  possible 
to  zoom  into  detail,  as  illustrated  schematically  in  Figure  6.1  if  a  recipi¬ 
ent  of  the  briefing  asks  questions.  Although  not  shown  here,  the  zooms 
can  provide  a  visual  explanation  of  results.  For  example,  if  someone 
asks  why  his  favorite  option  shows  up  as  red  in  Table  6.2,  a  zoom  to 
the  next  level  will  show  what  that  result  depends  on.  The  explanation 
may  be  that  while  several  of  the  necessary  conditions  are  met  (green), 
at  least  one  of  the  critical  conditions  is  not  (red).  Thus,  the  option  is 
ineffective.  For  example,  an  option  with  a  great  delivery  platform  and 
weapon  will  do  no  good  if  it  lacks  the  intelligence,  surveillance,  and 
reconnaissance  to  find  the  target.  Or,  an  option  may  work  well  unless 
the  adversary  adopts  a  countermeasure  that  intelligence  now  credits 
him  as  having  already  deployed. 

Often,  senior  leaders  have  too  little  time  to  delve  into  details,  but 
they  will  ask  selective  questions  because  of  curiosity,  prior  knowledge, 
the  importance  of  some  particular  judgment,  or  a  desire  to  test  the 
staff’s  mettle.  Deputies  and  senior  staff,  however,  often  demand  a  deep 


Table  6.2 

Portfolio-Analysis  Display  from  a  Prompt  Global  Strike  Study 


Mission 

Option 


Attack  Mobile 
ICBMs 


Mobile 
Missiles  A 


Mobile 
Missiles  B 


Attack  Terrorist  Leadership 
Meeting 

Terrorist  Terrorist 
Leadership  A  Leadership  B 


SOURCE:  Adapted  from  Davis  et  al.,  2008b. 


Attack  Hardened 
WMD  Facilities 


WMD  WMD  Program  Employ-  Cost 

Facilities  A  Facilities  B  Risk  ability  Risk  ($B) 


N.A. 

Low 

0 

Low 

Low 

10 

Low 

Low 

50 

Medium 

Low 

30 

Low 

Low 

60 

Medium 

Low 

40 

High 

Low 
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understanding  of  the  analysis:  It  is  their  responsibility  to  “scrub”  the 
work. 

At  the  bottom  of  Figure  6.1,  the  schematic  indicates  that  score- 
cards  only  go  so  far.  At  some  point,  addressing  issues  requires  using 
the  kinds  of  charts  more  familiar  in  systems  analysis — e.g.,  tradeoff 
curves. 

Significantly,  designing  analysis  to  permit  displays  such  as 
Table  6.2  and  zooming  leads  to  highly  disciplined  work  and  a  sense  of 
where  the  analysis  is  and  should  be  going.  Further,  if  the  form  of  pre¬ 
sentation  is  discussion  with  a  few  graphics,  rather  than  a  full-up  brief¬ 
ing,  the  presenter  will  be  able  to  answer  questions  verbally  drawing  on 
background  knowledge. 


Figure  6.1 

Zooming  into  Detail  for  Explanation 
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The  Economics  of  Portfolio  Analysis 

My  recommended  approach  to  portfolio  analysis  emphasizes  discuss¬ 
ing  issues  at  the  policy-scorecard  level,  as  in  the  section  above.  There 
should  be  no  early  attempt  to  consolidate  in  some  notional  overall 
“effectiveness”  because  the  measures  are  incommensurate3  and  one 
seldom  has  a  good  understanding  in  advance  of  their  relative  weights  (a 
linear  concept),  much  less  more  subtle  relationships.  Rather,  one  “dis¬ 
covers”  those  relationships  in  the  process  of  viewing  the  multicriterion 
results  and  contemplating  how  better  options  can  be  constructed  by 
adjustments  to  improve  results  for  some  objectives,  even  while  accept¬ 
ing  lesser  results  than  intended  by  others.  For  example,  a  policymaker 
might  come  into  a  discussion  believing  that  certain  goals  for  particular 
criteria  are  hard  and  fast,  but  might  then  realize  that  giving  no  atten¬ 
tion  to  another  objective  is  unacceptable.  To  do  better  may  require 
“paying  for  it”  by  reducing  goals  under  other  objectives.  The  same  kind 
of  iterative  “discovery”  of  values  should  be  familiar  to  the  reader  from 
long  experience  with  consumer-level  rating  systems.  We  do  not  usually 
want  to  just  accept  an  overall  judgment  of  some  consumer  magazine; 
instead,  we  want  to  know  how  the  options  fare  by  different  criteria. 

Discussions  with  policymakers  at  the  scorecard  level  can  be  both 
efficient  and  profound,  leading  to  better  articulation  of  what  will  be 
acceptable.  At  that  point,  analysts  can  do  some  “neatening  up,”  which  is 
particularly  helpful  when  refining  the  economic  analysis.  That  is,  ana¬ 
lysts  can  construct  overall  effectiveness  in  terms  of  the  effectiveness  of 
the  top-level  criteria,  doing  so  in  a  way  that  reflects  policymaker  intent 
and  values  (which  were  not  well  understood,  much  less  articulated, 
previously).  The  basis  is  then  laid  for  cost-effectiveness  calculations. 

Since  the  effectiveness  of  options  depends  on  numerous  uncer¬ 
tain  assumptions  and  judgments  on  which  there  may  be  disagreement 
among  policymakers,  however,  it  follows  that  cost-effectiveness  needs 
to  be  interpreted  with  some  sophistication.  If  important  assumptions 
and  disagreements  cluster,  then  it  is  possible  to  exploit  this  by  defin¬ 
ing  alternative  “strategic  perspectives,”  one  for  each  major  cluster  of 
assumptions,  values,  and  judgments.  In  a  debate  about  force  structure, 
an  example  would  be  that  some  policymakers  might  cluster  around 
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a  forward-leaning  military  posture  in  the  Asia-Pacific  region,  with 
investments  in  various  forces  and  weapon  systems  associated  with  the 
air-sea  battle  and  large-scale  deployment  of  prompt  long-range  strike 
systems.  Others  might  cluster  around  a  strategy  focused  more  on  con¬ 
trolling  sea  lanes  broadly  and  having  what  they  saw  as  less-provocative 
military  capabilities  while  maintaining  an  acceptable  military  balance. 
Still  others  might  cluster  around  a  strategy  giving  much  more  weight  to 
continued  counterinsurgency  and  counterterrorism  operations  in  the 
Middle  East,  Africa,  and  perhaps  elsewhere. 

Figure  6.2  illustrates  the  idea  notionally  using  just  two  perspec¬ 
tives.  The  two  curves  are  efficient  frontiers  for  Perspectives  I  and  II  (i.e., 
there  are  no  options  on  the  table  that  are  more  effective  at  a  given  cost 
or  less  expensive  at  a  given  effectiveness).  Each  perspective  looks  at  the 
range  of  increasingly  expensive  force  building  options,  but  evaluates 
them  differently. 

Those  holding  Perspective  I  might  see  all  the  options  on  the  table 
as  desirable  with  Option  A,  B,  C,  and  D  having  significantly  better 
effectiveness  (but  at  an  increasing  cost).  In  contrast,  those  with  Per¬ 
spective  II  might  see  little  or  no  value  for  going  beyond  Option  B. 

Figure  6.2 

Generalized  Efficient-Frontier  Depictions  for  Different  Perspectives 
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Moreover,  they  might  believe  that  Option  D,  although  more  expensive, 
would  be  downright  counterproductive.  To  use  the  examples  above, 
Perspective  I  might  be  calling  for  a  relatively  vigorous  buildup  of  offen¬ 
sive  options  in  the  Asia-Pacific  region,  while  Perspective  II  might  be 
doubtful  of  their  value  and  quite  negative  about  the  most  ambitious 
option  (D)  because  of  concerns  about  stimulating  arms  competition 
and  escalating  peacetime  tensions. 

The  example  is  contrived,  but  the  methodological  points  being 
made  are  valid: 

•  Analysis  should  recognize  that  cost-effectiveness  comparisons  can 
be  sensitive  to  strategic  assumptions  on  which  there  exists  sub¬ 
stantial  deep  uncertainty,  disagreement,  or  both. 

•  Cost-effectiveness  comparisons  should  routinely  highlight  the 
implications  of  both  uncertainty  in  the  usual  sense  (the  need  for 
more  information)  and  strategic  disagreement. 

•  Finally,  as  discussed  in  Chapter  Five,  when  selecting  candidate 
options  as  a  function  of  budget  to  be  allocated,  it  is  important 
to  keep  options  that  are  “near”  the  efficient  frontier  under  any 
of  the  salient  strategic  perspectives.  Otherwise,  potentially  attrac¬ 
tive  options  will  be  eliminated  prematurely  based  on  questionable 
assumptions. 


Tailoring  Portfolio  Analysis  to  Classes  of  Decision 

The  discussion  above  suggests  that  portfolio  analysis  can  be  presented 
so  as  to  be  relatively  simple  and  understandable  to  policymakers  “at  a 
glance,”  to  include  providing  visual  explanation  of  underlying  assump¬ 
tions  via  the  zooming  described  above.  Experienced  analysts,  however, 
will  suspect  that  this  is  too  good  to  be  true  because  the  issues  are  truly 
complicated.  The  resolution  of  this  apparent  contradiction  is  simple 
enough,  but  important: 

•  Portfolio  analysis  must  be  structured  differently  for  different  poli¬ 
cymaking  contexts. 
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In  some  cases,  this  can  simply  mean  tailoring  displays,  but  in 
other  cases  it  will  mean  structuring  the  underlying  analysis  differently. 
More  specifically,  policymakers  worrying  about  future  force  structure, 
future  force  capabilities,  how  best  to  balance  active  and  reserve  com¬ 
ponents,  or  how  best  to  deal  with  the  immediate  and  short-term  bud¬ 
gets,  are  working  different  problems.  They  need  to  see  different  “top- 
level  factors”  and  different  “explanations”  of  why  one  or  another  option 
looks  attractive  or  unattractive.  Ideally,  an  analysis  organization  will 
have  an  integrated  understanding  of  how  these  different  “views  of  the 
elephant”  relate  to  each  other  and  will  ensure  that  they  are  ultimately 
consistent.  However,  that  does  not  lend  itself  well  to  “simple”  models, 
“simple”  calculations,  and  “simple”  displays.  One  virtue  of  campaign 
models  is  that  building  and  using  them  establishes  expertise  that  can  in 
principle  be  harnessed  for  a  variety  of  simpler  and  more  focused  efforts. 
As  discussed  in  Chapters  Three  and  Four,  however,  this  often  does 
not  happen  and  managers  or  consumers  of  analysis  need  to  insist  that 
greater  priority  be  given  to  the  simpler  and  more  focused  efforts.  This 
has  many  implications  for  investment,  staffing,  and  terms  or  reference 
for  the  “analysis  campaigns”  that  I  have  discussed. 


Endnotes 

1  Hie  more  useful  methods  are  sometimes  referred  to  as  robust  optimization 
because  they  consider  the  consequences  of  relaxing  alleged  requirements  and  con¬ 
straints.  They  are  often  used  iteratively  within  an  evolving  decision  process.  For  some 
examples  of  optimization  as  part  of  strategic  planning,  see  Helms,  2012,  Hoehl  and 
Scales,  2011,  and  Davis  et  ah,  2008b,  which  were  discussed  at  a  May  2013  Military 
Operations  Research  Society  meeting.  For  a  discussion  of  good  modern  practice  in 
optimization,  including  robustness  methods,  see  Brown  and  Rosenthal,  2008.  See 
also  recent  work  specifically  addressing  uncertainty  (Chow,  2013). 

2  The  methods  began  in  research  with  strategic-level  support  of  the  first  Quadren¬ 
nial  Defense  Review  (Davis,  Kugler,  and  Hillestad,  1997).  Years  later,  the  methods 
were  substantially  enhanced  in  work  for  the  Director  of  the  Missile  Defense  Agency 
and  two  Under  Secretaries  of  Defense  for  Acquisition,  Technology,  and  Togistics. 
Some  documents  are  in  the  public  domain  (Davis  et  ah,  2008a,  2008b;  Davis  and 
Dreyer,  2009). 
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3  James  Schlesinger  made  similar  observations  decades  ago  (Schlesinger,  1974).  To 
be  sure,  many  very  useful  studies  have  combined  criteria  into  a  single  scale  (Ham¬ 
mond,  Keeney,  and  Raiffa,  2002;  Keeney  and  RaifFa,  1976;  Keeney,  1992). 


CHAPTER  SEVEN 

Conclusions 


In  this  monograph,  I  have  tried  to  sketch  ways  in  which  defense  analy¬ 
sis  can  better  serve  the  needs  of  policymakers,  especially  in  the  cur¬ 
rent  period  of  austerity.  I  have  not  discussed  the  substantive  issues  of 
the  day,  but  rather  paradigms  and  methods  for  the  management  of 
analysis.  The  emphasis  has  been  on  the  subset  of  analysis  for  strategic 
planning  that  requires  more  than  collecting  data,  thinking  clearly,  and 
creating  good  charts  and  point  papers. 

The  claim  of  the  monograph  is  that  “big  studies”  should  be  con¬ 
ceived  in  terms  of  analysis  campaigns.  Analysts  should  worry  from  the 
outset  about  what  analytic  information  policymakers  need  and  what 
final  summary  conclusions  might  look  like — but  without  presuppos¬ 
ing  a  particular  answer  or  slant.  They  should  be  sensitive  to  uncer¬ 
tainties  and  determined  to  distill  the  implications  of  those  uncertain¬ 
ties  usefully.  This  will  often  mean  identifying  affordable  hedges  that 
allow  for  later  adaptations.  Helping  the  organization  plan  for  flexibil¬ 
ity,  adaptiveness,  and  robustness  is  part  of  the  analyst’s  job,  as  is  dem¬ 
onstrating  that  it  is  possible  to  achieve  objectives  while  keeping  within 
budget  constraints  (assuming  that  it  is  indeed  possible). 

The  campaign  should  be  constructed  to  provide  the  underpin¬ 
nings  for  the  answer  (actually  many  answers,  in  different  levels  of 
detail).  The  campaign  may  include  a  mix  of  simple  and  not-so-simple 
analysis,  drawing  systematically  on  models  of  different  types,  gaming, 
operations  planning  and  insights  from  operators,  historical  data,  and 
other  sources  of  information.  It  includes  technology-push  and  demand- 
pull  contrasts.  The  options  considered  should  go  beyond  those  offered 
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up  by  the  various  advocates  or  programs.  Further,  the  options  should 
often  include  “paying”  for  additions  with  subtractions.  And,  especially 
in  the  present  period,  options  should  seek  substantial  cost  savings — 
for  their  own  sake  given  budget  constraints,  and  also  to  make  room  for 
initiatives  important  to  the  future.  This  may  require  new  technology, 
new  concepts  of  organization,  and  new  organizational  structures — 
suggestions  for  which  may  not  be  forthcoming  through  usual  channels. 

Analysts  should  often  construct  composite  options  suitable 
to  the  policy  decisions  at  issue,  using  a  joint  perspective  with  com¬ 
binations  for  which  there  may  be  no  initial  advocates.  Preliminary 
screening  should  identify  composite  options  that  are  potentially  cost- 
effective  when  viewed  from  a  variety  of  perspectives  and  over  the  range 
of  uncertainties.  They  should  then  use  portfolio-analysis  methods  to 
structure  discussion  of  options  according  to  multiple  measures  of  merit 
corresponding  to  the  various  relevant  objectives,  including  forms  of  risk 
reduction.  Analysts  should  be  prepared  immediately  to  explain  results 
analytically  in  more  detail  than  summary  depictions.  Even  summary 
depictions  should  preemptively  show  how  results  depend  on  combina¬ 
tions  of  major  assumptions  and  on  one’s  “strategic  perspective.” 

Finally,  Table  7.1  contrasts  what  this  monograph  suggests  (right 
column)  to  what  sometimes  occurs  in  current  analysis.  The  word  “some¬ 
times”  applies  because  analysis  and  analysis  organizations  vary  sub¬ 
stantially.  Many  past  studies  are  consistent  with  the  attributes  shown. 
Managers  of  analysis  will  recognize  many  of  the  admonitions  and  note 
their  attempts  to  do  something  similar.  Nonetheless,  in  aggregate,  the 
table  is  intended  to  point  a  way  forward  that  is  significantly  at  odds 
with  much  current  practice. 
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Table  7.1 
Contrasts 


Attribute 

To  Be  Avoided 

To  Be  Encouraged 

Interaction  with 
policymakers 

Initial  tasking;  final  briefing 
and  report 

Structured  discussion,  iteration,  discussion, 

.  .  .;  policymakers  are  part  of  the  analysis; 
follow-up  report 

Character  of  output 

Answers 

Information,  framework,  explanation,  and 
story  to  aid  choice  and  subsequent  commu¬ 
nication,  persuasion,  and  implementation 

Explanation 

Reference  to  underlying 
model  runs 

An  understandable  top-level  story  with 
understandable  zooms  into  layers  of  detail 
as  needed 

Uncertainties 

Identification  of  assumptions 

Preemptively  showing  consequences  of  all 
major  assumptions  or  decisions,  depending 
on  simpler  analysis  methods  to  do  para¬ 
metric  work;  emphasis  on  hedging 

Origin  of  test  cases 

High-level  committees  for 
big-picture;  lower-level  staff 
work  on  critical  parameters, 
sometimes  with  log-rolling 

Analyst-designed  candidate  scenarios 
and  parametrics  followed  by  iterative 
discussions  with  policymakers  to  tune 
scenarios  and  uncertainty  ranges,  resulting 
in  better  policymaker  understanding 
of  implications;  result:  establishing 
meaningful  requirements  with  good  test 
cases 

Assessment  of 
options 

Optimization  of  some  overall 
"utility" 

Multiobjective  scorecards,  with  zooms  for 
explanation;  emphasis  on  assessing  results 
in  terms  of  flexibility,  adaptiveness,  and 
robustness  (what  others  call  support  for 
robust  decisionmaking  or  planning  for 
agility) 

Reports  to 
policymakers 

Capability  shortfalls  and 
requests  for  additional  funds 
to  fill  them 

Consequences  of  budget  cuts 

Options  and  evaluations  providing 
different  portfolio  balances,  for  each  of 
several  budget  levels;  tradeoff  curves;  etc. 

Options  that  include  major  future¬ 
leaning  changes  exploiting  technology, 
new  concepts  of  operation,  and  new 
organizational  approaches 

Response  to  "what- 
if?" 

Requests  for  more  time 

Presentation  of  "capabilities  analysis," 
which  preemptively  addresses  the  what-ifs, 
perhaps  with  real-time  "zooming"  routine 
quick-turn  analysis  when  needed 

Nature  of  tools  used 

Standardized  complex  models, 
standardized  precise  scenarios 
and  data 

Mix  of  simple  and  complex  models  with  the 
simple  ones  providing  structure  of  story 
and  natural  metrics;  routine  parametrics 
that  treat  standard  cases  merely  as  points 
of  comparison;  also,  use  of  gaming, 
historical  analysis,  expert  judgment  from 
operators,  etc. 

APPENDIX  A 


Subjective  Data  on  Past  Studies 


Table  A.l  characterizes  subjectively  a  number  of  past  studies  by  using 
scores  from  1  to  5  to  indicate  how  strongly  attributes  in  the  top  row 
were  present.  The  cases  included  the  damage-limiting  study1  and 
STRAT  X  study  from  the  1960s,2  the  surface-to-air  missile  upgrade 
studies  from  the  period  1967-1972, 3  the  POMCUS  study  of  the  early 
1970s,4  unpublished  studies  (1978-1981)  leading  to  the  Rapid  Deploy¬ 
ment  Joint  Task  Force  that  became  U.S.  Central  Command,5  some  Star 
Wars  studies  of  the  1980s,6  studies  of  conventional  forces  in  Europe 
(1985-1993), 7  the  Deep-Attack  Weapon  Mix  Study  (1990s),8  various 
Halt  studies  (1996-2002), 9  a  detailed  evaluation  of  the  Comanche 
weapon  system  (1990s),10  a  recent  tanker  study  (2000s),11  and  social- 
science-intensive  studies  of  the  2010s.12 
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Table  A.1 

Subjective  Assessment  of  Attributes  in  Selected  Past  Studies 


Case 

Strategy 

Concepts  of 
Operations, 
Doctrine 

Capability 

Analysis 

"Hard": 

Physics, 

Technology 

"Soft": 

Social 

Science 

Cost- 

Effectiveness 

Analysis 

"Simple" 

Modeling 

Complex 

Modeling 

Damage-limiting  (1960s) 

3 

3 

5 

3 

1 

5 

5 

3 

STRAT-X  (1960s) 

5 

5 

5 

5 

1 

5 

5 

4 

Surface-to-air  missile 
upgrade  (1967-1972) 

1 

5 

5 

5 

1 

2 

5 

5 

POMCUS  (1970s) 

2 

3 

3 

2 

1 

5 

5 

2 

Rapid  Deployment  Joint  Task 
Force  (1979-1981) 

5 

5 

3 

2 

1 

3 

5 

3 

Star  Wars,  arms  control 
(1980s) 

3 

5 

5 

5 

1 

5 

3 

5 

Conventional  forces  in 

Europe  (1985-1993) 

3 

5 

3 

2 

1 

5 

5 

5 

Deep-attack  mix  (1990s) 

2 

5 

5 

3 

1 

5 

2 

5 

Halt  studies  (1996-2002) 

3 

3 

5 

2 

1 

5 

5 

4 

Comanche  simulation  study 
(1990s) 

1 

5 

5 

5 

1 

2 

1 

5 

Tanker  study  (2000s) 

1 

5 

3 

2 

1 

5 

1 

5 

Qualitative  counterterrorism 
modeling  (2010s) 

3 

1 

1 

1 

5 

1 

5 

1 
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Endnotes 

1  The  study  has  apparently  not  been  released,  but  is  described  in  a  memoir  (Kent  et 
ah,  2008)  and  an  earlier  paper  (Kent,  2002),  which  notes  “these  computations  were 
made  on  slide  rules  and  Friden  calculators  and  that  the  allocations  were  determined 
by  measuring  the  slope  of  lines,  by  triangles  and  rulers,  on  utility  plots  fastened  to  a 
large  table.” 

2  Fragments  of  the  STRAT-X  study  have  been  released  and  some  broad  discussion 
is  available  (Grier,  2010). 

3  See  an  excellent  account  by  a  retired  Central  Intelligence  Agency  official  (Ste¬ 
vens,  2013). 

4  The  unpublished  POMCUS  studies  were  done  by  OSD  (Program  Analysis  and 
Evaluation),  but  key  elements  are  described  in  Congressional  Budget  Office  docu¬ 
ments  and  a  book  by  Richard  Kugler  (2006,  p.  564). 

3  These  studies  began  with  the  now-declassified  “Wolfowitz  study”  (OSD  [PA&E], 
1979),  the  completion  of  which  I  led  under  Paul  D.  Wolfowitz.  That  was  followed 
by  informal,  unpublished  analyses  in  1980-1981  that  relied  on  qualitative  con¬ 
siderations,  facts,  logic,  and  hand  calculations,  supplemented  by  sophisticated 
strategic-mobility  models  (OSD  [PA&E],  1981)  and  insights  from  computer-assisted 
war-gaming  in  the  Joint  Staff. 

6  This  refers  to  separate  classified  RAND  studies  on  the  Strategic  Defense  Initia¬ 
tive  accomplished  by  James  A.  Thomson,  Russell  D.  Shaver,  Maurice  Eisenstein, 
Richard  Darilek,  and  others.  Some  of  the  most  important  analysis  required  math¬ 
ematical  prowess,  but  not  large  computer  models.  Other  parts  required  complex 
computerized  kinematic  modeling  led  by  Michael  Miller.  See,  e.g.,  Wilkening  et  al., 
1989. 

7  There  were  many  studies  on  the  conventional  balance  in  Europe  and  related 
arms  control.  One  of  the  most  influential  highlighted  a  simple  story  (Thomson  and 
Gantz,  1987).  Others  went  well  beyond  that  but  also  sought  to  be  readily  under¬ 
stood  without  mathematics  (Davis,  1988a,  1988b;  Thomson,  1988).  Underlying 
this  work,  however,  was  substantial  research  using  the  Joint  Integrated  Contingency 
Model. 

8  The  Deep  Attack  Weapons  Mix  Study  was  led  by  George  Kolezar,  James  Bexfield 
(analysis),  and  Richard  Nelson  (cost)  of  the  Institute  for  Defense  Analyses  and  a 
DoD  committee  with  representatives  from  numerous  offices.  The  work  was  based 
on  the  tactical  warfare  campaign  model  and  an  optimization  model.  The  study’s 
methodology  is  described  in  a  published  case  study  (Bexfield,  2001). 

9  A  number  of  interdiction  or  “halt”  studies  were  accomplished  at  RAND  and  else¬ 
where,  some  with  the  Joint  Integrated  Contingency  Model  or  comparable  campaign 
models  and  some  with  much  simpler  models.  See  also  Appendix  F. 
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10  The  RAND  work  on  Comanche  was  led  by  Monti  Callero  in  a  classified  study 
that  used  entity-level  simulation  of  both  ground  forces  and  air  forces. 

11  Michael  Kennedy  led  several  tanker-related  studies,  some  with  intense  congres¬ 
sional  interest.  In  part  because  of  controversy,  they  were  extraordinarily  rigorous 
and  relied  on  detailed  computer  models,  sharply  defined  scenarios  and  databases, 
careful  cost  analysis,  and  optimization  (Kennedy  et  al.,  2006). 

12  A  number  of  such  studies  have  been  conducted  in  recent  years  by  colleagues. 
Some,  led  by  Kim  Cragin,  Todd  Helmus,  and  Brian  Jackson  have  been  classified. 
They  involved  significant  data  collection  and  analysis  for  the  war  zones,  but  did 
not  use  computer  modeling.  Much  of  the  work  used  qualitative  factor-tree  models 
(Downes-Martin,  2013).  Recent  work  has  demonstrated  how  such  qualitative 
models  can  be  implemented  as  uncertainty-sensitive  computational  models  (Davis 
and  O’Mahony,  2013).  Such  models  are  not  for  point  predictions  but  rather  for 
exploratory  analysis  to  see  what  combinations  of  factors  need  to  be  favorable  for  suc¬ 
cess  to  be  reasonably  plausible. 


APPENDIX  B 


Capabilities-Based  Planning 


This  appendix  describes  analytical  aspects  of  the  history  of  capabilities- 
based  planning  and  discusses  the  confusion  that  has  often  arisen  about 
how  it  compares  to  threat-based  planning  (ultimately,  a  non-issue  as 
discussed  at  the  end).  What  is  described  is  what  I  see  as  the  “good” 
kind  of  capabilities-based  planning,  which  is  rather  different  from  the 
way  it  is  sometimes  perceived  because  of  implementation  issues  (e.g., 
with  complex  bureaucratic  processes  and,  ironically,  with  overuse  of 
point  scenarios). 


History 

The  Early  Period,  1961-1980 

The  planning  methods  of  the  1960s  and  1970s  are  often  said  to  have 
been  based  on  threat-based  planning.1  That  label,  however,  conflated  to 
very  different  issues:  (1)  highlighting  a  particular  named  adversary  and 
(2)  how  analysis  and  planning  dealt  with  scenario  uncertainties  and 
how-much-is-enough  issues.  Cold  War  defense  budgets  benefited  from 
the  United  States  having  the  Soviet  Union  as  a  known,  formidable,  and 
dangerous  adversary.  Highlighting  this  “capital-T  threat”  was  a  main 
feature  of  how  defense  programs  were  presented  and  discussed.  How¬ 
ever,  as  illustrated  below,  the  stronger  elements  of  defense  planning 
were  capabilities-based,  even  if  that  is  not  widely  recognized. 

Strategic  Nuclear  Forces.  Since  1961,  force  planning  for  long- 
range  nuclear  forces  has  used  constructs  that  help  in  planning  under 
uncertainty.  By  the  mid-1970s,  when  I  was  immersed  in  such  work  as 
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part  of  OSD  (Program  Analysis  and  Evaluation)^  strategic  forces  divi¬ 
sion,  analysts  varied  whether  U.S.  and  Soviet  forces  were  on  day-to-day 
alert  or  on  generated  alert,  which  side  would  attack  first,  and  whether 
the  sides  would  use  launch  under  attack.  We  considered  counterforce, 
countermilitary,  and  countervalue  attack  options  for  both  sides.*  We 
varied  such  technical  assumptions  as  the  projected  accuracy  of  future 
Soviet  missiles,  whether  the  sides  would  try  to  limit  civilian  deaths 
with  air  bursts  rather  than  ground  bursts,  and  weapon-system  reliabili¬ 
ties.  Overall,  we  used  dozens  ^/"scenarios/ cases,  none  being  regarded 
as  a  meaningful  best  estimate  or  base  case.  Rather,  force  planning 
depended  on  understanding  the  totality.  We  certainly  had  in  mind 
a  specific  adversary,  a  capital-T  threat,  but  in  other  respects  this  was 
capabilities-based  planning.  To  be  sure,  public  debate  often  revolved 
around  a  specific  scenario,  such  as  a  surprise  Soviet  counterforce  first 
strike  followed  by  U.S.  response  in  kind.  That  was  useful  as  one  bound¬ 
ing  case.  Other  scenarios,  however,  were  essential  for  net  assessment, 
understanding  the  military  balance,  or  understanding  a  more  plausible 
play-out  of  nuclear  war. 

Strategic  Mobility.  Another  important  class  of  force-planning 
analysis  involved  strategic  mobility — long-range  airlift,  sealift,  and  the 
use  of  prepositioned  material.  DoD’s  analysis  and  advocacy  was  cru¬ 
cial  because  mobility  forces  lacked  the  natural  champions  enjoyed  by 
new  fighter  aircraft,  carrier  battle  groups,  or  tanks.  The  analysis  used 
standard  scenarios  as  test  cases  and  for  presentations,  but  policymak¬ 
ers  always  valued  the  flexibility  of  strategic  mobility  forces  demon¬ 
strated  by  working  scenarios  requiring  deployments  to  various  theaters 
under  varied  conditions.  Again,  then,  much  of  the  underlying  analysis 
was  capabilities-based  even  though  external  discussion  was  typically 
oriented  toward  the  massive  Soviet  threat  and  goals  were  set  accord¬ 
ingly.  The  conclusion  of  a  major  study  from  this  period  was  expressed 
in  capability  terms:  a  requirement  for  airlift  capability  of  66  million 
ton  miles  per  day.2  This  “requirement”  was  just  a  reasonable  judgment 
based  on  considering  numerous  plausible  scenarios  seen  as  test  cases.3 


Counterforce,  countermilitary,  and  countervalue  targeting  refer  to  attacking  just  the 
adversary’s  nuclear  forces,  its  broader  military  forces,  or  its  urban-industrial  base. 
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Conventional  Forces.  Conventional  force  planning  was  seemingly 
more  threat-based.  The  primary  adversary  (the  Warsaw  Pact)  was  speci¬ 
fied  and  force  planning  in  the  1960s  and  1970s  used  stylized  idealiza¬ 
tions  of  threat  buildups  to  estimate  force  needs.  This  said,  higher-level 
planning  was  more  capabilities-oriented  if  one  went  beyond  superfici¬ 
alities.  Even  as  early  as  McNamara’s  first  years,  OSD  looked  at  a  wide 
range  of  possible  conflicts  and  what  capabilities  would  be  needed  to 
deter  or  deal  with  them.  As  it  happened,  it  proved  possible  to  focus 
force-sizing  on  a  combination  of  the  Soviet  threat,  the  North  Korean 
threat,  and  a  possible  “miscellaneous”  smaller  war.  Higher-level  analy¬ 
sis  over  the  next  20  years  often  referred  to  planning  for  2-1/2,  2,  or 
1-1/2  wars.  The  higher-level  analysis  was  often  quite  simple  (even  sim¬ 
plistic),  counting  major  formations  (divisions,  wings,  aircraft  carrier 
groups,  Marine  amphibious  groups)  and  using  rules-of-thumb  to  esti¬ 
mate  force  needs.4 

Planning  in  the  services  and  combatant  commands  was  more 
detailed — not  only  for  current  operations  planning,  but  also  for  build¬ 
ing  service  programs  (Program  Objective  Memoranda).  One  unfor¬ 
tunate  aspect  of  this  was  a  tendency  over  time  to  focus  on  rather  pre¬ 
cise  notions  of  how  war  would  be  conducted.  It  sometimes  seemed  as 
though  the  memoranda  were  being  developed  to  optimize  against  the 
official  war  scenario.  This  style  of  planning  identified  the  Soviets  as 
threat  and  used  well-specified  scenarios  (e.g.,  scenarios  indicating  the 
time  lines,  focus  of  attacks,  and  duration  of  war).  It  was  probably  what 
people  mean  when  thinking  about  classic  threat-based  planning. 

Even  so,  some  strategic  analysis  considered  nonstandard  cases.  The 
examples  with  which  I  am  most  personally  familiar  involved  Central 
Region  scenarios  with  creative  Soviet  strategies  and  Southwest  Asia 
scenarios.5  Perhaps  the  most  influential  example  was  the  “Wolfowitz 
study”  of  1979,  aptly  titled  Capabilities  for  Limited  Contingencies  in 
the  Persian  Gulf 6  The  study  had  been  commissioned  by  Presidential 
Directive  18  (August  24,  1977)  because  of  concerns  about  possible 
U.S.  military  weakness  in  the  Persian  Gulf  region.  The  study  had  an 
inauspicious  reception,  but  follow-on  work  led,  in  1980-1983,  to  the 
strategy,  force-building  programs,  and  command  changes  associated 
with  the  Rapid  Deployment  Joint  Task  Force,  which  evolved  in  1983 
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into  U.S.  Central  Command.7  Remarkably,  programs  initiated  in  the 
Carter  administration  were  embraced  and  expanded  in  the  Reagan 
administration. 

Although  clearly  “capabilities  analysis,”  this  work  looked  closely 
at  the  most  obvious  big  threats  to  the  Persian  Gulf  region,  notably  the 
prospect  that  Iraq  would  invade  or  coerce  Kuwait  and  the  prospect 
that  the  Soviet  Union  might  invade  Iran  and,  subsequently,  the  entire 
region.  These  were  not  regarded  as  likely,  but  as  worrisome  possibilities 
to  deter  by  assuring  that  the  United  States  had  suitable  military  capa¬ 
bilities.  Other  possible  contingencies  were  considered  in  lesser  detail. 
Overall,  the  emphasis  was  on  viewing  the  whole  and  understanding 
what  military  capabilities  would  be  especially  valuable  and  affordable. 

The  Late  Cold  War  Period,  1981-1992 

DoD-level  force-planning  work  did  not  change  greatly  between  1981 
and  1990 — even  though  the  primary  force-building  scenario  written 
in  1980-1981  made  no  sense  by  the  mid  to  late  1980s.8  One  change 
that  did  occur,  however,  was  that  related  analysis  in  both  OSD  and  the 
Joint  Staff  came  to  depend  more  on  large  campaign  models,  detailed 
specification  of  the  scenarios,  and  related  databases.  The  campaign 
models  allowed  richer  and  more  subtle  analysis  of  some  issues  and  the 
standardization  enabled  better  joint  analysis,  since  those  in  the  various 
services  could  use  the  same  information  and  indeed  had  to  do  so  when 
advocating  for  programs.  However,  the  ascendance  of  this  approach 
correlated  with  reduced  nimbleness  and  a  trees-rather-than-forest  per¬ 
spective.  It  was  during  this  time  that  threat-based  planning  began  to  be 
increasingly  associated  with  standardized  models,  standardized  scenar¬ 
ios,  complicated  databases,  lengthy  coordination  processes,  and  every¬ 
thing  except  broad  assumption-varying  analysis. 

As  of  1990  and  the  fading  of  the  Soviet  Union,  planning  changed, 
as  exhibited  by  the  1990  “Base  Force”  discussion  of  the  George  H.W. 
Bush  administration.9  General  Colin  Powell,  Chairman  of  the  Joint 
Chiefs,  recognized  that  it  was  important  to  base  force  levels  on  general 
U.S.  interests  and  conflict  possibilities  rather  than  on  highly  specific 
threats  and  threat  scenarios.10  Referring  to  a  change  from  a  threat- 
based  force  to  a  threat-  and  capability-based  force,  Powell  argued: 
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We  might  not  face  the  old  threat  from  the  Soviet  Union,  I  said, 
but  we  had  to  maintain  certain  fundamental  capabilities.  For 
example,  we  might  no  longer  have  a  specific  airlift  requirement 
to  move  X  million  tons  of  materiel  to  Europe  to  meet  a  poten¬ 
tial  Soviet  invasion.  But  we  still  needed  the  capability  to  move 
huge  stores  to  unpredictable  troubles  spots  around  the  world.  We 
might  no  longer  face  the  8th  Guards  Army  across  the  Fulda  Gap, 
but  we  still  needed  the  capability  to  project  power  elsewhere.  I 
proposed  forces  capable  of  performing  four  basic  missions:  one 
to  fight  across  the  Atlantic;  a  second  to  fight  across  the  Pacific;  a 
contingency  force  at  home  to  be  deployed  rapidly  to  hot  spots,  as 
we  did  in  Panama;  and  a  reduced  but  still  vital  nuclear  force  to 
deter  nuclear  adversaries. 

Powell’s  claim  was  understandably  not  convincing  to  everyone.  He 
did  not  include  a  compelling  reason  for  why  the  United  States  needed 
the  capabilities  he  referred  to.  It  was  obvious  to  him  after  a  career  of 
unexpected  military  events,  but  not  to  skeptics  (especially  those  from 
a  different  political  party  than  the  then-current  administration).  Even 
if  one  accepted  the  argument  that  the  United  States  still  needed  these 
capabilities,  the  question  of  how  much  was  needed  remained.  Surely, 
it  must  be  less  without  the  Soviet  Union.  This  criticism  was  unfair 
because,  at  Powell’s  urging,  the  Base  Force  was  to  be  a  good  25  percent 
smaller  than  the  previous  force.  That  said,  by  Powell’s  own  description 
of  his  personally  drafted  “Strategic  Overview  1994,”  his  figures  for  the 
Base  Force  as  of  1989  were  intuitively  driven  estimates.11  They  were  not 
obviously  rooted  in  analysis,  although — behind  the  scenes — his  J-8 
organization  did  a  good  deal  of  forward-thinking  analysis. 

Powell  and  his  deputies,  then,  were  attempting  to  institutionalize 
capabilities-based  planning,  to  include  dealing  with  uncertainty  and 
making  difficult  choices  under  budget  constraints.  This  approach  was 
consistent  with  the  preference  of  the  Under  Secretary  of  Defense  for 
Policy,  Paul  Wolfowitz.  The  approach  also  included  explicit  hedging, 
such  as  the  ability  to  reverse  trends  if  need  be12 — very  much  in  the 
spirit  of  this  monograph.  It  was  only  when  convinced  of  the  plan’s 
reversibility  that  Secretary  Dick  Cheney  came  to  support  it. 
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This  moving  away  from  threat-based  planning  did  not  mean 
ignoring  the  Soviet  Union  or  thinking  only  in  generic  terms.  Rather, 
it  had  to  do  with  recognizing  that  many  possible  sources  of  crisis 
or  conflict  existed,  including  the  Soviet  Union,  and  that  how  crisis 
would  arise  and  what  U.S.  forces  would  need  to  do  was  inherently 
uncertain — not  unbounded,  but  uncertain.  The  specificity  of  what  had 
become  known  as  threat-based  planning  was  inappropriate.  Also,  had 
threat-based  planning  been  used,  the  logic  would  have  called  for  even 
larger  force  reductions  because  the  primary  threat  was  falling  apart, 
not  just  becoming  a  bit  weaker. 

Post-Cold  War  Period,  1993-2000 

In  1993,  the  newly  elected  President  Bill  Clinton  appointed  Les  Aspin 
as  Secretary  of  Defense.  Aspin  had  been  Chairman  of  the  House  Armed 
Services  Committee  and  was  on  record  as  opposed  to  capabilities- 
based  planning,  insisting  that  threat-based  planning  was  necessary. 
One  of  his  statements  was  widely  publicized: 

It  is  critical  to  identify  threats  to  U.S.  interests  that  are  suffi¬ 
ciently  important  that  Americans  would  consider  the  use  of  force 
to  secure  them.13 

Much  has  been  made  of  this,  but  Aspin  did  not  mean  what  many 
have  claimed.  Aspin  was  making  a  judgment  about  what  was  necessary 
to  sell  the  defense  program  to  Congress  and  the  public,  rather  than 
commenting  on  what  concepts  should  underlie  the  program.  By  dint 
of  long  experience  on  the  committee,  Aspin  understood  the  need  for 
capabilities.  He  was  aware  of  uncertainties  and  emphasized  in  his  study 
called  the  Bottom-Up  Review  (similar  to  the  Quadrennial  Defense 
Review)  that  the  planning  scenarios  were  for  force  sizing:  How  forces 
would  actually  be  used  was  another  matter.  The  Bottom-Up  Review14 
also  took  a  building-block  approach  to  force  planning  with  the  intent 
of  being  prepared  for  diverse  challenges,  as  emphasized  in  capabilities- 
based  planning.  In  a  section  on  “Scenarios  as  Planning  Tools,”  the 
Bottom-Up  Review  observed:15 
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History  suggests  that  we  most  often  deter  the  conflicts  that  we 
plan  for  and  actually  fight  the  ones  that  we  do  not  anticipate. 


For  planning  and  assessment  purposes,  we  have  selected  two 
illustrative  scenarios  that  are  both  plausible  and  posit  demands 
characteristic  of  those  that  could  be  posed  by  conflicts  with  other 
adversaries.  Figure  4  [not  shown]  displays  the  scenarios  and  their 
relationship  to  planning  for  force  employment  across  a  range  of 
potential  conflicts.  While  a  number  of  scenarios  were  examined, 
the  two  that  we  focused  on  most  closely .  . .  envisioned  aggression 
by  a  remilitarized  Iraq  .  .  .  and  by  North  Korea.  .  .  . 

Despite  the  scrapping  about  planning-method  labels,  then,  the 
Bottom-Up  Review  actually  built  on  and  was  largely  consistent  with 
the  Base  Force  concept  of  the  Bush  administration  although  it  went 
10-15  percent  further  in  force  reductions.16  As  I  wrote  at  the  time, 
the  Bottom-Up  Review  framework  had  problems,  such  as  its  reliance 
on  old-fashioned  building-block  forces  (divisions,  wings,  and  aircraft 
carrier  groups)  dating  back  to  World  War  II,  relying  excessively  on  the 
North  Korean  and  Iraqi  threats  for  force  sizing,  and  not  being  forward¬ 
leaning,17  but  it  was  not  nearly  so  “threat-based”  as  it  seemed  casually. 
Moreover,  by  the  mid-1990s,  DoD  began  expanding  substantially  the 
number  and  classes  of  scenario  that  inside-the-Pentagon  planning  was 
to  address — i.e.,  it  was  moving  even  further  toward  capabilities-based 
planning,  whether  or  not  the  term  was  in  vogue. 

The  next  Quadrennial  Defense  Review  under  Secretary  William 
Cohen  introduced  a  new  strategic  framework  for  planning  {Shape, 
Respond,  Prepare  Now).18  It  also  had  a  chapter  on  “Transformation,” 
although  not  mandating  changes  in  forces.  It  preserved  the  force-sizing 
requirement  to  be  able  to  fight  two  major  theater  wars  simultaneously, 
but  Secretary  Cohen  was  adamant  about  distinguishing  between  the 
strategic  framework  and  that  force-sizing  consideration.  That  is,  the 
capability  to  fight  and  win  two  overlapping  wars  was  considered  nec¬ 
essary  for  the  strategy,  but  was  not  itself  the  strategy.  He  emphasized 
having  forces  that  could  respond  quickly  and  decisively  to  the  full  spec- 
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trum  of  crises,  and  to  be  prepared  for  “wild  card”  scenarios.19  At  the 
same  time,  he  directed  a  number  of  additional  cost-cutting  measures. 

By  the  late  1980s,  there  was  recognition  that  the  requirement  to 
be  able  to  fight  two  major  theater  wars,  although  explicitly  intended  to 
be  only  for  force  sizing,  was  being  misinterpreted  as  though  it  were  spe¬ 
cifically  about  Iraq  and  North  Korea,  and  thereby  causing  problems.  A 
change  of  criterion  was  arguably  needed  to  rebroaden  thinking  as  had 
been  intended  in  1997.20 

The  Early  New-Century  Period,  2001-2010 

In  2001,  Secretary  Donald  Rumsfeld  established  the  capabilities- 
based  approach  as  a  new  management  framework,  seeing  it  as  a  way 
to  manage  strategic  risks  in  a  highly  uncertain  world.21  He  built  on  de 
facto  developments  from  the  Clinton  administration  such  as  assessing 
more  diverse  threats,  but  was  much  more  vigorous  in  pursuing  trans¬ 
formation  and  bringing  about  management  change.  He  is  often  quoted 
for  saying  that  a  capabilities-based  model  is  one  that 

focuses  more  on  how  an  adversary  might  fight  rather  than  specifi¬ 
cally  whom  the  adversary  might  be  or  where  a  war  might  occur. 

Unfortunately,  many  people  reading  parts  of  the  2002  Quadrennial 
Defense  Review  believed  that  the  Secretary  was  precluding  discussion 
of  specific  potential  adversaries.  For  a  time,  officials  within  DoD  lit¬ 
erally  insisted  that  briefings  not  refer  to  named  threats.  Perhaps  this 
misimpression  was  exacerbated  by  a  more  colloquial  article  in  which 
Rumsfeld  said 

It’s  like  dealing  with  burglars:  You  cannot  possibly  know  who 
wants  to  break  into  your  home,  or  when.  But  you  do  know  how 
they  might  try  to  get  in.  You  know  they  might  try  to  pick  your 
lock,  so  you  need  a  good,  solid,  dead  bolt  on  your  front  door.  You 
know  they  might  try  breaking  through  a  window,  so  you  need  a 
good  alarm.  You  know  it  is  better  to  stop  them  before  they  get  in, 
so  you  need  a  police  force  to  patrol  the  neighborhood  and  keep 
bad  guys  off  the  streets.  And  you  know  that  a  big  German  Shep¬ 
herd  doesn’t  hurt,  either. 
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To  head  off  confusion,  he  might  have  added  an  aside  after  the  “You 
cannot  possibly  know”  sentence:  “Well,  perhaps  you  have  some  idea; 
you  might  worry  about  a  particular  suspicious  neighbor.  However,  you 
still  don’t  know  the  when  or  how;  you  can’t  simply  schedule  a  time 
to  be  at  home  sitting  with  a  gun  at  the  window  you  expect  him  to 
enter.”  Does  anyone  doubt  that  Rumsfeld  would  have  said  something 
like  this  if  his  rhetorical  language  “You  cannot  possibly  know”  had 
been  challenged? 

If  one  goes  back  to  the  original  document,  it  turns  out  that  Rums¬ 
feld’s  language  in  the  Quadrennial  Defense  Review  did  not  actually 
have  the  narrow  interpretation  that  would  reject  considering  particular 
threats.  On  the  very  first  page  (p.  iii)  he  says  (referring  to  the  attack  of 
September  11  as  backdrop): 

the  attack  .  .  .  highlights  a  fundamental  condition  of  our  cir¬ 
cumstances;  we  cannot  and  will  not  know  precisely  where  and 
when  America’s  interests  will  be  threatened,  when  America  will 
come  under  attack,  or  when  Americans  might  die  as  the  result 
of  aggression.  We  can  be  clear  about  trends,  but  uncertain  about 
events.  We  can  identify  threats,  but  cannot  know  when  or  where 
America  or  its  friends  will  be  attacked.  We  should  try  mightily  to 
avoid  surprise,  but  we  must  also  learn  to  expect  it.  .  .  .  Adapting 
to  surprise — adapting  quickly  and  decisively — must  therefore  be 
a  condition  of  planning. 

Also,  when  describing  the  capabilities-based  approach  in  more  detail, 
he  says  (p.  13): 

That  concept  [the  capabilities-based  approach]  reflects  the  fact 
that  the  United  States  cannot  know  with  confidence  what  nation, 
combination  of  nations,  or  non-state  actor  will  pose  threats.  .  .  . 

It  is  possible,  however,  to  anticipate  the  capabilities  that  an  adver¬ 
sary  might  employ  to  coerce  its  neighbors,  deter  the  United 
States  from  acting  in  defense  of  its  allies  and  friends,  or  directly 
attack  the  United  States  or  its  deployed  forces.  A  capabilities- 
based  model — one  that  focuses  more  on  how  an  adversary  might 
fight  than  who  the  adversary  might  be  and  where  a  war  might 
occur — broadens  the  strategic  perspective.  It  requires  identifying 
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capabilities  that  U.S.  military  forces  will  need  to  deter  and  defeat 
adversaries  who  will  rely  on  surprise,  deception,  and  asymmetric 
warfare  to  achieve  their  objectives.  .  .  . 

These  longer  passages  certainly  include  worrying  about  particular 
adversaries  (how  could  anyone  imagine  otherwise?).  Rumsfeld,  how¬ 
ever,  was  correctly  asserting  that  all  details  of  scenario  were  uncertain 
(recall  the  scenario-space  depiction  of  Figure  3.7),  thereby  implying 
that  specific  threat  scenarios  should  not  be  the  focus  of  planning  even 
if  their  concreteness  makes  them  compelling.22 

When  introducing  capabilities-based  planning,  Rumsfeld  also  had 
strong  ideas  relating  to  organization  and  management.  He  was  quite 
troubled  about  how  jointness  in  planning  was  attempted  only  after 
the  services  had  put  together  their  programs  independently.  He  recog¬ 
nized  that  concepts  and  capabilities  should  be  conceived  and  developed 
in  joint  terms  from  the  outset.  This  emphasis  on  joint  thinking  was 
reflected  in  a  major  implementation  study  (the  “Aldridge  study”)23  that 
led  to  substantial  reorganization  of  processes — mostly  for  good  but 
with  some  stumbles  as  well  (especially  overly  burdensome  processes).  In 
ongoing  work,  officials  created  an  analytic  agenda  that  was  to  empha¬ 
size,  slightly  paraphrased,  “deeper  understanding,  collaborative  devel¬ 
opment  of  data,  more  robust  treatment  of  uncertainty,  responsiveness, 
an  understanding  of  differences  among  competing  studies,  support  for 
Combatant  Commander  planning  activities,  and  support  of  future- 
force  development.”21 

The  Recent  Period,  2009- 

Tfie  2010  Quadrennial  Defense  Review  issued  under  Secretary  of 
Defense  Robert  Gates  further  embellished  the  risk-management  struc¬ 
ture  introduced  in  2001  but  was  largely  seen  as  a  continuation  of  policy. 
It  continued  the  emphasis  on  capabilities-based  thinking25: 

In  short,  U.S.  forces  today  .  .  .  can  be  plausibly  challenged  by 
a  range  of  threats  that  extend  far  beyond  the  familiar  “major 
regional  conflicts”  that  have  dominated  U.S.  planning  since  the 
end  of  the  Cold  War.  .  .  .  the  wars  we  fight  are  seldom  the  wars 
that  we  would  have  planned  ...  it  is  no  longer  appropriate  to 
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speak  of  “major  regional  conflicts”  as  the  sole  or  even  the  primary 
template  for  sizing,  shaping,  and  evaluating  U.S.  forces.  Rather, 

U.S.  forces  must  be  prepared  to  conduct  a  wide  variety  of  mis¬ 
sions  under  a  range  of  different  circumstances. . . .  The  QDR  thus 
employed  several  scenario  combinations  to  represent  the  range  of 
likely  and/or  significant  challenges  anticipated  in  the  future  and 
tested  its  force  capacity  against  them. 

The  entire  section  is  well  worth  reading  for  those  interested  in 
the  issue  of  capabilities-based  planning  versus  threat-based  planning, 
and  in  defense  analysis  generally.  The  discussion  is  clearly  intended  to 
be  “analytic.”  The  scenarios  used  for  the  Quadrennial  Defense  Review 
were  not  just  “reasonable,”  but  rather  were  chosen  to  be  “test  cases.” 
There  was  no  intent  to  forecast.  The  authors  of  the  Review  were  explor¬ 
ing  the  landscape  of  challenges.  At  some  point,  DoD  would  issue  plan¬ 
ning  scenarios  to  serve  as  “requirements,”  but  not  because  the  scenarios 
were  thought  to  be  correct. 

Somewhat  later,  the  administration  of  President  Barack  Obama 
announced  major  changes  of  strategy  that  included  a  pivot  toward 
Asia,  better  described  as  as  a  matter  of  rebalancing — reducing  empha¬ 
sis  on  large-scale  counterinsurgency  operations  and  giving  renewed 
attention  to  the  the  Asia-Pacific  region.  Not  much  is  publicly  available 
about  what  new  planning  scenarios  will  be  like,  but  it  can  be  assumed 
that  those  will  be  changed  to  match  the  times.  As  discussed  in  a  2011 
report,  attention  should  shift  more  to  worrying  about  balance-of-power 
issues,  deterring  small-scale  aggression,  avoiding  crises,  and  managing 
them  if  they  arise,  than  about  building  forces  for  a  particular  scenario 
such  as  the  battle  over  the  Taiwan  Straits.26  With  respect  to  North 
Korea,  it  now  seems  more  important — given  South  Korea’s  substantial 
military  capability — to  worry  about  specific  issues  such  as  potential 
coercion  attempts  or  missile  attacks  than  about  all-out  classic  invasion. 
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Definitions  and  Contrasts:  Real  and  Imagined 

Early  Connotations 

The  2002  Quadrennial  Defense  Review  did  not  include  definitions  of 
capabilities-based  or  threat-based  planning,  or  describe  consequences 
for  analysis.  At  the  time,  the  best  summary  of  what  people  probably 
believed  the  terms  meant  was  in  a  balanced  article  by  John  Troxell  of 
the  Army  War  College,  who  addressed  the  period  from  the  1950s  to 
2001  and  pointed  out  overlaps  and  misunderstandings  related  to  both 
terms.27  He  summarized  comparisons  as  follows: 

Threat-based  planning: 

This  methodology  is  preeminent  when  threats  to  U.S.  interests 
are  easily  recognized  and  identified.  The  task  for  the  planner  is  to 
postulate  a  reasonable  scenario,  or  a  specific  military  contingency, 
then  determine  the  amount  of  force  needed  to  prevail  in  that  sce¬ 
nario.  The  approach  lends  itself  to  dynamic  and  static  modeling 
and  provides  quantifiable  rationale  for  the  recommended  force 
structure.  .  .  .  The  logic  ...  is  very  compelling  and  greatly  facili¬ 
tates  accomplishing  the  planner’s  third  task — convincing  the 
public  and  Congress.28 

Capabilities-based  planning: 

Capabilities-based  planning  is  most  in  vogue  when  threats  to 
U.S.  interests  are  multifaceted  and  uncertain,  and  do  not  lend 
themselves  to  single  point  scenario-based  analysis.  Instead  of 
focusing  on  one  or  more  specific  opponents,  the  planner  applies 
a  liberal  dose  of  military  judgment  to  determine  the  approximate 
mix  of  required  military  capabilities.  Capabilities-based  plan¬ 
ners  claim  to  focus  on  objectives  rather  than  scenarios.  A  major 
problem  planners  have  with  this  approach  is  convincing  Con¬ 
gress  that  military  judgment  has  established  the  proper  linkage 
between  the  recommended  force  and  the  uncertain  geostrategic 
environment.29 
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Tightening  Up  Concepts 

My  own  definition  was  published  in  a  2002  monograph  (Davis,  2002, 
p.  xi),  but  stemmed  from  a  decade  of  previous  work.30 

Capabilities-based  planning  is  planning  under  uncertainty  to 
provide  capabilities  for  a  wide  range  of  modern-day  challenges 
and  circumstances  while  working  within  an  economic  framework 
that  necessitates  choice. 

The  intent  was  to  convey  in  plain  language  the  common-sense  nature 
of  the  approach.  Planning  should  confront  uncertainty  and,  recogniz¬ 
ing  the  inability  to  forecast  accurately  what  challenges  will  arise  or, 
significantly,  the  detailed  circumstances,  should  provide  “capabilities” 
able  to  deal  with  what  does  arise.  Capabilities,  however,  do  not  come 
free  and  choices  must  be  made  so  as  to  live  within  a  budget.  In  many 
ways,  the  intent  was  to  refocus  analysis  on  enduring  principles  such  as 
described  in  Chapter  Two. 

Although  never  adopted  formally,  this  definition,  or  minor  vari¬ 
ants,  has  been  used  and  cited  widely  in  DoD  work,31  including  in  a 
white  paper  from  the  Joint  Staff.32  Other  agencies  such  as  the  Govern¬ 
ment  Accountability  Office,  Federal  Emergency  Management  Agency, 
and  the  Department  of  Homeland  Security  have  used  it  or  a  variant.33 
It  has  also  been  cited  prominently  by  the  National  Academy  (National 
Research  Council,  2005);  an  international  defense  planning  group 
from  the  United  States,  United  Kingdom,  Australia,  and  New  Zea¬ 
land34;  and  the  scholarly  literature.35 

The  definitional  issues  run  deeper.  What  does  “capability”  mean? 
In  day-to-day  language,  the  word  has  two  different  meanings.36  One 
refers  to  “general  wherewithal”  (as  in,  “we  haven’t  actually  planned  for 
that,  but  we  could  deal  with  it  because  we  have  the  requisite  general 
capabilities”).  In  a  variant,  a  Secretary  of  Defense  or  Chairman  may 
refer  to  capability  as  being  measured  by  force  structure,  modernization, 
readiness,  and  sustainability.37  The  other  connotation  has  to  do  with 
being  able  truly  to  accomplish  a  specified  mission.38  If  the  President 
asks  a  commander,  “Do  you  have  the  capability  to  do  this  mission?”  a 
positive  answer  should  mean  that  the  mission  can  actually  be  accom- 
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plished  successfully,  implying  adequacy  of  physical  equipment,  person¬ 
nel,  training,  plans,  support  forces,  etc.  This  meaning  is  what  led  to  the 
sometimes-cited  but  unfortunate  DoD  definition:39 

Capability:  the  ability  to  achieve  a  desired  effect  under  specified 
standards  and  conditions  through  combinations  of  means  and 
ways  to  perform  a  set  of  tasks 

One  commenter  wryly  observed,  “For  an  effort  aimed  at  clarifica¬ 
tion,  this  must  be  the  lexicographical  equivalent  of  destroying  a  village 
to  save  it.”40  Still,  the  definition’s  motivation  is  understandable.  If  DoD 
establishes  a  requirement  for  a  capability,  the  services  and  combatant 
commands  can  promise  to  achieve  it  only  if  they  know  for  what  they 
are  responsible.  That  is,  there  is  a  contract  involved.  Unfortunately,  this 
definition  is  easily  misread  to  undercut  the  core  concept  of  capabilities- 
based  planning.  The  solution  is  straightforward: 

•  The  apparent  contradiction  can  be  reconciled  by  interpreting 
“under  specified  standards  and  conditions”  to  mean  “under  a 
specified  range  of  standards  and  conditions.” 

An  analogy  would  be  for  an  aircraft  designer  to  promise  capabil¬ 
ity  to  operate  within  a  flight  envelope.  Commanders  and  other  “oper¬ 
ators”  are  intuitively  inclined  to  this  interpretation,  but  contractors 
building  a  piece  of  equipment  might  have  an  incentive  to  interpret 
matters  more  narrowly.  More  insidiously,  that  is  often  so  as  well  for 
analysts  who  work  with  big  models  and  painfully  coordinated  data¬ 
bases.  In  my  view,  analysis  organizations  have  often  undercut  the  kind 
of  capabilities-based  planning  that  was  intended  by  DoD  leadership. 

Comparisons 

What  about  the  definition  of  threat-based  planning?  How  do  they 
relate?  My  answer  is  as  it  was  a  decade  ago:41 

Capabilities-based  planning  is  often  contrasted  in  discussion 
and  articles  with  “threat-based  planning,”  which  is  confusing 
because  capabilities-based  planning  is  also  very  much  concerned 
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about  threats.  No  one  seriously  proposes  that  the  Department 
of  Defense  should  spend  nearly  $400  billion  per  year  for  general 
insurance  against  the  abstract  possibility  that  some  threat  might 
conceivably  arise  somewhere,  sometime — especially  when  threats 
currently  exist  and  other  potential  challenges  can  be  seen  on  the 
horizon. 


It  follows  that  the  correct  contrast  is  not  with  “threat-based  plan¬ 
ning”  as  that  phrase  is  interpreted  literally,  but  rather  with  depen¬ 
dence  on  a  specific  bounding  threat  as  represented  by  one  or  a 
very  few  point  scenarios. 

To  elaborate,  Table  B.l  characterizes  variants  of  capabilities-based 
and  threat-based  planning  by  various  attributes.  It  includes  extreme 
and  more  reasonable  versions,  the  latter  being  in  the  middle  rows,  par¬ 
ticularly  those  shaded  in  blue.  TBP+  corresponds  with  what  is  favored 
by  people  who  say  they  prefer  threat-based  planning  or  a  hybrid  of 
TBD  and  CBP.  Such  people  do  value  what  I  have  called  FARness  and 
they  do  make  clear  distinctions  between  analytic  threats  for  the  sake  of 
power  balances,  for  example,  and  capital-T  threats  that  are  to  be  greatly 
feared.  They  see  planning  scenarios  as  useful  instruments,  not  fore¬ 
casts.  Such  scenarios  can  be  forcing  functions  of  change,  can  provide 
illustrative  details  to  support  a  host  of  activities  from  training  to  exer¬ 
cising,  and  can  definitely  be  used  in  analysis  as  the  basis  for  choice  and 
judgments  about  how  much  is  enough.  They  do  not  just  make  up  such 
scenarios  out  of  thin  air  or  slavishly  follow  some  Intelligence  Com¬ 
munity  projection.  Further,  if  asked,  they  are  eager  for  analysis  that 
examines  variations  of  assumptions  and  leads  to  robust  conclusions.  In 
my  own  mind,  these  modern-day  defenders  of  what  they  call  TBD  are 
actually  practitioners  of  CBP.  However,  by  clinging  to  the  terminology 
of  TBD,  by  often  putting  too  much  emphasis  on  the  planning-scenario 
cases,  and  by  not  more  forcefully  advocating — and  even  demanding — 
more  exploratory  work,  they  differ  significantly  in  degree  from  propo¬ 
nents  of  CBP.  It  seems  clear  to  me,  if  not  to  the  reader,  that  CBP  as 
defined  here,  is  the  “right”  way  to  do  analysis,  and  is  rooted  in  deep 
principles.  TBP,  in  contrast  (even  TBP+)  will  continue  to  take  point 


Table  B.1 

A  Comparison  of  Planning  Methods,  Actual  and  Contrived 


Breadth  of 
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•  • 
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•  • 

•  • 

•  • 

Nutty  CBP 

•  • 

•  • 

•  • 

NOTES:  TBP+:  threat-based  planning  at  the  strategic  level,  especially  in  communication,  but  with  broader  and  deeper  work  beneath 
the  surface,  in  mission-level  analysis  and  sometimes  in  strategic-level  side  rooms.  By  "nutty"  I  mean  to  suggest  than  no  one  sensible 
would — if  pushed — define  the  methods  in  this  way. 
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scenarios  far  too  seriously,  suppressing  important  uncertainties  in  the 
process  and  undervaluing  flexibility,  adaptiveness,  and  robustness.  The 
organizational  and  psychological  pressures  for  doing  so  are  very  strong. 

Ultimately,  labels  are  not  especially  important  in  themselves, 
especially  when  their  meanings  are  so  blurred.  Thus,  one  solution  if 
the  controversy  continues  would  include  sharpening  the  definition  of 
CBP  and  embracing  it  fully,  noting  that  it  contains  the  good  features  of 
TBD.  Another  possible  solution  could  take  the  form  of  some  new  label 
purporting  to  combine  the  best  features  of  both. 


Communicating  and  Convincing  Others 

A  remaining  issue  is  the  one  flagged  initially  by  Secretary  Aspin  and 
mentioned  from  time  to  time  in  many  settings.42  What  approach  is 
better  for  “selling”  the  defense  program?  The  question  is  uncomfort¬ 
able  because  analysis  should  focus  on  objective  assessments,  not  politics 
or  salesmanship.  Nonetheless,  policy  analysis  must  concern  itself  with 
communication  and  implementation. 

Secretary  Aspin  may  well  have  been  correct  with  respect  to  what 
proved  convincing  to  the  House  Armed  Services  Committee  during 
his  tenure,  but  he  was  not  describing  a  law  of  nature.  It  is  true  that  the 
U.S.  public  supports  increases  in  the  defense  budget  during  periods  of 
perceived  threat  and  may  demand  decreases  in  periods  of  low  perceived 
threat.  It  is  not  obvious,  however,  that  the  public  is  affected  by  the 
esoterica  of  planning  methodologies.  Influential  members  of  Congress 
and  their  staffs  might  be,  but  it  is  worth  noting  that  the  U.S.  Navy 
and  U.S.  Marine  Corps  have  done  well  over  the  years  with  Congress 
even  though  their  force  structures  have  never  been  threat-based.  More 
dramatically,  the  Intelligence  Community’s  programs  have  flourished 
over  the  last  decade  even  though  they  are  capabilities-based  rather  than 
built  around  particular  scenarios. 
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Principles,  Independent  of  Labels 

It  is  important  to  move  beyond  labels  and  chronic  misinterpretations. 

Some  suggestions  are  listed  here: 

•  The  underlying  core  analysis  and  force  planning  should  be 
capabilities-based  because  of  massive  uncertainty,  but  should 
also  (rather  obviously)  consider  specific  threats  that  can  be 
identified — whether  to  understand  the  potential  challenges  or 
assess  how  much  capability  is  enough  when  the  budget  dictates 
the  need  for  choice. 

•  Scenarios  developed  for  force-sizing  explanations  must  not  be 
allowed  to  undercut  preparing  forces  for  flexibility,  adaptiveness, 
and  robustness.  Nor  should  they  be  taken  as  forecasts. 

•  Clear  distinctions  should  be  maintained  between  test-case  sce¬ 
narios  conceived  for  assessing  balances  of  power  and  scenarios 
conceived  for  operational  planning  or  more  general  contingency 
preparations.  There  may  be  overlaps,  but  the  differences  are  and 
should  be  substantial. 

•  How  the  merits  of  a  proposed  defense  program  are  communi¬ 
cated  most  effectively  will  depend  on  the  era.  Sometimes,  it  is 
most  effective  to  dramatize  particular  threats;43  other  times,  it  is 
most  effective  to  promote  broad  capabilities  and  illustrate  their 
value  for  many  possible  purposes.  In  either  case,  however,  both 
the  underlying  analysis  and  the  programs  themselves  should  be 
designed  for  flexibility,  adaptiveness,  and  robustness. 
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Other  Approaches  to  Risk  Management 


DoD's  Risk-Management  Language  in  Policy  Documents 

The  main  text  treats  reducing  risk  as  just  another  kind  of  objective. 
Other  approaches  are  possible.  In  particular,  if  one  considers  defense 
planning  to  be  basically  an  exercise  in  risk  management,  then  every¬ 
thing  becomes  a  risk.  That  is,  all  of  DoD’s  functions  and  objectives  can 
be  expressed  in  terms  of  reducing  one  or  another  kind  of  risk.  That  is  the 
tack  taken  in  the  2001,  2006,  and  2010  Quadrennial  Defense  Reviews, 
which  used  a  risk-management  framework  that  evolved  somewhat  over 
time.  The  most  recent  version  distinguishes  four  types  of  risk:1 2 3 

1.  Operational  risk:  the  ability  of  the  current  force  to  execute  strat¬ 
egy  successfully  within  acceptable  human,  materiel,  financial, 
and  strategic  costs.  Consideration  of  operational  risk  requires 
assessing  the  Department’s  ability  to  execute  current,  planned, 
and  contingency  operations  in  the  near  term 

2.  Force  management  risk:  our  ability  to  recruit,  retain,  train,  edu¬ 
cate,  and  equip  the  All-Volunteer  Force,  and  to  sustain  its  readi¬ 
ness  and  morale.  This  requires  the  Department  to  examine  its 
ability  to  provide  trained  and  ready  personnel  in  the  near  term, 
midterm,  and  long  term. 

3.  Institutional  risk:  the  capacity  of  management  and  business 
practices  to  plan  for,  enable,  and  support  the  execution  of  DoD 
missions.  It  encompasses  the  ability  to  develop  effective  and  effi¬ 
cient  organizations  and  processes  over  the  near  term,  midterm, 
and  long  term. 
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4.  Future  challenges  risk:  the  Department’s  capacity  to  execute 
future  missions  successfully,  and  to  hedge  against  shocks.  Here 
most  consideration  is  given  to  the  Department’s  ability  to  held 
superior  capabilities  and  sufficient  capacity  to  deter/defeat 
emerging  threats  in  the  midterm  and  long  term. 

These  categories  are  clearly  appropriate  focuses  of  attention.  Note, 
however,  that  the  definitions  refer  to  “risk”  with  such  words  as  “abil¬ 
ity”  and  “capacity.”  The  categories  actually  refer  to  objectives  and  the 
“risks”  have  to  do  with  inability  to  meet  those  objectives.  Thus,  the 
framework  is  actually  part  of  an  objectives-based  framework  in  dis¬ 
guise.  When  the  Clinton  administration  referred  to  “Prepare  Now”  as 
part  of  its  strategy,  it  was  addressing  “Future  Challenges  Risk”  in  dif¬ 
ferent  language.  Preferences  vary. 

DoD’s  current  risk  framework  is  especially  useful  at  the  top  level 
for  distinguishing  among  organizationally  and  managerially  different 
problems  so  as  to  help  define  the  responsibilities  charged  to  varied  parts 
of  the  organization.  It  also  distinguishes  between  the  near  and  long 
term  and  between  the  more  visible  aspects  of  capability  (e.g.,  forces 
and  force  capabilities)  and  underlying  considerations  such  as  the  qual¬ 
ity  and  training  of  personnel  and  the  effectiveness  and  efficiency  of 
organizational  behavior. 


Other  Government  Uses  of  "Risk" 

DoD  is  a  very  large  organization  and  the  word  “risk”  has  different 
meanings  in  its  different  parts.  For  example,  a  recent  report  by  OSD 
means  by  risk  the  potential  for  a  program  to  come  in  over  cost  or  over 
schedule  or  to  underperform.2  In  contrast,  within  a  specific  research 
and  development  program  the  risk  of  most  concern  might  be  that  a 
technology  cannot  be  made  to  work.  In  recruiting  activities,  the  risk 
may  be  that  economic  developments  will  cause  the  supply  of  high-qual¬ 
ity  recruits  to  dry  up  before  (if  at  all)  Congress  is  willing  to  increase 
incentive  pay. 
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Moving  outside  DoD  for  further  examples,  the  Department  of 
Homeland  Security  has  used  a  risk-based  approach  to  analysis  in  which 
various  categories  of  risk  are  combined  into  an  overall  metric.  Program 
alternatives  can  then  be  compared  as  to  how  much  they  “reduce  risk,” 
i.e.,  how  much  they  improve  the  risk  metric.  As  discussed  in  the  main 
text,  there  are  major  shortcomings  to  that  approach,  as  noted  also  by 
a  National  Academy  report  (National  Research  Council,  2010).  The 
risk  metric  is  so  aggregated  as  to  have  little  meaning  and  comparing 
options  by  small  changes  in  this  dubious  risk  metric  obscures  impor¬ 
tant  issues.  As  discussed  in  Chapter  Three,  it  is  important  to  maintain 
visibility  across  multiple  top-level  criteria.  Department  of  Homeland 
Security  leadership  is  aware  of  the  shortcomings  of  the  methodology 
and  is  attempting  to  deal  with  them  in  a  variety  of  ways. 

The  conclusion  from  this  should  be,  simply,  that 

•  Referring  without  explanation  to  “risk”  is  not  useful.  It  is  neces¬ 
sary  to  attach  modifiers  specifying  which  kind  of  risk  is  being 
referred  to.* 

•  In  many  contexts,  it  is  important  to  have  an  explicit  taxonomy  of 
risks  to  ensure  both  good  communication  and  appropriate  atten¬ 
tion  to  all  of  them. 

•  In  many  contexts,  it  is  necessary  to  specify  concretely  how  the  par¬ 
ticular  risk  being  discussed  is  measured  or  calculated.  Anything 
less  that  this  specificity  will  likely  result  in  misunderstanding. 

What  follows  illustrates  some  of  these  points  in  more  detail. 


Normal  Risk  Versus  Risk  from  Deep  Uncertainties 

Strategic  planning  must  distinguish  clearly  between  what  might  be 
called  “normal”  variation  of  results,  some  of  which  is  downside  risk, 
and  the  risk  that  results  will  be  far  worse  than  expected  because  or 


This  theme  is  also  emphasized  in  recent  work  by  Air  Force  A-9,  which  suggests  a 
common  standard  for  discussing  risk.3 
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unplanned-for  events,  a  misunderstanding  of  the  problem,  or  some¬ 
thing  else. 

Figure  C.l  illustrates  the  issue  schematically  by  plotting  the 
cumulative  probability  that  outcome  will  be  worse  than  Outcome 
X,  versus  Outcome  X  itself.  The  expected  result  based  on  a  planning 
case  is  5,  which  is  “ok.”  The  anticipated  range  is  4-6  because  of  vari¬ 
ous  details  and  random  factors,  but  anything  in  the  range  is  accept¬ 
able.  There  is  some  possibility — even  for  this  planning  case — of  worse 
results.  This  is  the  “tail  effect”  for  the  planning  case.  However,  the 
much  larger  risk  is  that  the  basic  planning  case  itself  is  fundamentally 
wrong.  Although  the  unexpected  cases  have  no  well-specified  probabil¬ 
ity  distribution,  the  red  dashed  curve  indicates  a  rough  assessment  that 
in  aggregate  such  poorly  understood  or  appreciated  cases,  part  of  deep 
uncertainty,  add  considerably  to  the  risks.  Some  people  refer  to  these 
as  “black  swan”  risks. 


Figure  C.l 

Distinguishing  Normal  Risks  and  Those  from  Deep  Uncertainty 
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Interestingly,  in  strategic  planning  it  is  often  possible  to  recog¬ 
nize  the  unexpected  cases  as  hypothetical  possibilities,  but  to  see  them 
individually  as  quite  improbable.1 * * 4  Unfortunately,  if  a  number  of  such 
possibilities  exist,  then  it  may  be  fairly  likely  that  at  least  one  of  them 
will  occur.  As  an  example,  given  six  hypothetical  bad  cases,  each  with 
only  a  10  percent  probability,  one  has  a  roughly  50/50  chance  that 
at  least  one  of  them  will  occur.  This  is  why  good  engineers  strive  for 
exceedingly  high  reliability  rates  on  system  components  with  many 
components. 

Because  really  bad  cases  are  so  often  quite  different  in  kind  from 
marginally  adverse  versions  of  nominal  cases,  it  is  useful  to  preserve 
the  term  “risk”  for  referring  to  them  rather  than  normal  but  adverse 
variations.  If  instead  we  use  the  word  risk  for  what  amount  to  the  small 
tail  effects  of  the  normal  case,  we  will  have  squandered  an  important 
word  in  the  English  language.  Further,  our  statement  about  risk  may 
be  quite  misleading.  The  suggestions  for  strategic  planning,  then,  ared 

•  Use  “risk”  to  refer  to  the  possibility  of  outcomes  significantly  worse 
than  anticipated,  which  will  often  correspond  to  effects  of  deep 
uncertainty. 

•  Use  “normal  risk”  when  referring  to  more  marginal  worse-than- 
expected  outcomes. 

This  might  suggest  a  display  such  as  in  Table  C.l  for  comparing 
options.  All  of  the  options  have  the  same  “expected”  consequence  and 
the  same  normal  range  of  consequences.  Option  2,  however,  builds  in 
the  possibility  of  a  somewhat  better  result  without  taking  additional 
risks  (maybe  it  includes  a  small  investment  in  research  and  develop¬ 
ment  with  a  very  high  upside  potential).  In  contrast,  Option  3  builds 
in  the  possibility  of  a  great  outcome  (1),  but  at  the  expense  of  intro- 


1  In  other  domains  of  risk  analysis,  the  phenomena  may  be  well  understood  and  charac- 

terizable  by  non-Gaussian  probability  distributions  with  “thick  tails.”  In  such  a  case,  it  is 

appropriate  to  equate  “risk”  with  the  consequence  of  those  thick  tails. 5  Invoking  deep  uncer¬ 

tainty  and  black  swans  is  unnecessary.  However,  if  the  phenomenon  itself  is  changing  (as 
with  climate  change  affecting  the  frequency  and  ferocity  of  storms),  then  historical  statistics 
are  unreliable  and  the  concepts  of  deep  uncertainty  do  apply. 
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Table  C.1 

Distinguishing  Normal  and  Deeper  Risks 


Option 

Expected 
Effectiveness 
and  Normal 
Range 

Best-Case 

Effectiveness 

(Upside 

Potential) 

Worst-Case 
Effectiveness 
(Risk  or 

Downside  Risk) 

1 

5  (4-6) 

6 

4 

2 

5  (4-6) 

7 

4 

3 

5  (4-6) 

10 

0 

ducing  substantial  risk.  An  operational  military  example  might  be  an 
option  having  all  forces  charge  forward  into  battle  as  rapidly  as  pos¬ 
sible,  hoping  to  utterly  rout  and  destroy  the  enemy,  but  at  the  price  of 
having  no  reserves.  The  absence  of  reserves  could  be  disastrous  in  the 
event  of  a  surprise  enemy  maneuver.  In  the  realm  of  force  planning  a 
real-world  example  might  be  the  Future  Combat  System  of  the  early 
2000s.  The  program  “bet  the  farm,”  so  to  speak,  on  a  combination  of 
immature  technology,  undeveloped  operational  concepts,  and  a  devel¬ 
opment  program  managed  by  a  commercial  contractor.  It  was  a  truly 
revolutionary  concept  with  what  seemed  to  some  to  have  a  very  rosy 
upside  potential,  but  admittedly  significant  risks  that  could  hardly  be 
called  normal.  The  program  failed.  Although  many  elements  of  the 
program  undoubtedly  had  value  that  will  pay  off  in  other  ways,  the 
outright  losses  amounted  to  many  billions  of  dollars. 


Distinguishing  Among  Sources  of  Risk 

Both  normal  risks  and  those  associated  with  deep  uncertainty  have  dif¬ 
ferent  origins  and  different  characters.  To  illustrate  how  taxonomies  of 
risk  can  be  useful,  let  us  consider  a  particular  study6  that  looked  at  alter¬ 
native  ways  to  develop  conventional  prompt  global  strike.  A  subsequent 
National  Academy  study  drew  on  this  work  but  included  additional 
options  and  reflected  further  debate  and  analysis  (National  Academy 
of  Sciences,  2008).  The  options  included,  e.g.,  depending  on  forward- 
deployed  aircraft,  a  new  conventional  warhead  and  delivery  system  for 
Trident  missiles,  a  new  intercontinental  ballistic  missile,  and  other  pos- 
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sibilities.  Each  such  option  would  have  a  certain  estimated  effective¬ 
ness,  if  developed  and  deployed  successfully.  Shortcomings  in  those 
measures  of  effectiveness  were  not  considered  “risks,”  but,  rather,  just 
capability  gaps.  For  example,  some  options  would  not  provide  enough 
firepower  to  damage  a  hard  deeply  buried  facility  for  weapons  of  mass 
destruction.  Others  would  not  have  the  ability  to  find  mobile  missiles. 
Again,  these  were  shortcomings,  not  “risks.”  In  contemplating  what 
risks  to  consider,  the  RAND  study  developed  the  taxonomy  in  Figure 
C.2,  ascribing  risks  of  various  types  for  each  type  of  mission  for  which 
the  option  was  being  tested.  The  left-most  branch  in  the  figure  was 
attacking  mobile  missiles  such  as  mobile  intercontinental  ballistic  mis¬ 
siles.  We  saw  a  risk  that  such  a  program  would  be  terminated  during 
development  because  of  a  political-strategic  judgment  by  Congress  or 
a  next  President  that  this  was  not  an  appropriate  mission  or  not  a  mis¬ 
sion  of  consequence.  For  some  options,  we  also  saw  substantial  techni¬ 
cal  and  programmatic  risks  because  the  option  required  feasible  but 
cutting-edge  technology  that  had  not  yet  been  demonstrated  (discussed 

Figure  C.2 

An  Illustrative  Taxonomy  of  Risks  for  Options  in  a  Particular  Study 
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extensively  in  the  National  Academy  study).  The  technology  might 
simply  fail  (an  unexpected  possibility),  or  it  might  just  take  longer 
and  cost  more  (normal  risk).  The  possibility  also  existed  that  even  if 
the  option  was  successfully  developed,  the  relevant  systems  might  not 
be  available  when  needed  because  of  being  maldeployed  or  otherwise 
engaged.  If  the  capability  were  actually  employed,  there  would  be  a  risk 
of  collateral  damage  or  erroneous  targeting.  There  were  also  other  risks, 
such  as  the  potential  that  employing  this  particular  option  would  cause 
an  unintended  escalation  (perhaps  because  it  would  overfly  countries 
that  would  mistake  its  purpose  or  perhaps  because  the  targeted  country 
would  choose  to  retaliate  in  a  different  and  less  limited  way).  The  point 
here  is  not  whether  these  were  the  “correct”  risks,  but  rather  that  there 
were  so  many  to  consider  even  in  an  illustrative  study.  Again,  discus¬ 
sion  of  risks  should  include  adjectives  distinguishing  among  them. 

Significantly,  the  most  important  class  of  risk  was  not  even  shown 
in  the  taxonomy  because  it  was  treated  in  a  different  way.  When  evalu¬ 
ating  the  options  against  the  several  missions,  we  used  two  test-case 
scenarios  for  each.  For  the  mission  of  attacking  mobile  missiles,  we 
considered  what  amounted  to  a  nonresponsive  threat  and  a  responsive 
threat — i.e.,  a  threat  as  was  currently  being  projected  by  intelligence 
and  one  that  would,  in  addition,  adopt  countermeasures.  Although 
we  treated  the  responsive-threat  case  as  merely  a  test  scenario  without 
labeling  it  as  “risk,”  the  plausibility  of  a  responsive  threat  is  so  high  that 
a  policymaker  would  certainly  consider  it  a  risky  proposition  to  build 
an  expensive  system  that  might  be  totally  ineffective. 

To  complete  this  example,  note  that  someone  working  on  cost- 
schedule  risk  might  do  a  superb  and  precise  job  with  empirical  data 
and  sophisticated  statistical  analysis.  He  or  she  might  report  the  risk  of 
being  more  than  10  percent  over  cost  or  more  than  two  years  behind 
schedule  as  less  than  25  percent,  a  rather  precise  figure.  In  contrast, 
there  is  no  way  to  “calculate”  objectively  the  many  other  large  risks, 
such  as  that  effective  countermeasures  would  be  deployed  or  that  use 
of  the  option  would  trigger  escalation.  Those  risks  have  to  be  estimated 
subjectively  and  qualitatively — in  some  cases  by  policymakers  them¬ 
selves.  The  National  Academy  study  had  strong  but  subjective  conclu¬ 
sions  based  on  long  experience  with  science,  technology  and  weapon- 
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system  developments.  It  identified  some  of  the  options  as  high  risk  if  it 
was  necessary  to  achieve  deployment  within  a  few  years. 


Common  Problems  in  Risk  Analysis 

Since  many  approaches  to  risk  analysis  exist  and  many  types  of  risk 
exist  to  be  analyzed,  it  is  worth  mentioning  some  common  sources  of 
confusion  and  trouble. 

•  When  an  organization  receives  less  than  it  requested  in  the  budget, 
it  may  warn  that  the  result  is  increasing  risks.  Such  a  claim  might 
or  might  not  be  accurate. 

-  Perhaps  the  decision  was  to  forgo  an  objective.  Inability  of  an 
option  to  achieve  something  that  is  no  longer  an  objective  is 
not  usefully  described  as  risk,  even  if  the  decision  to  forgo  the 
objective  is  indeed  risky  (as  when  Great  Britain  pulled  back 
from  East  of  Suez). 

-  Perhaps  the  decision  drew  on  other  organizations’  experience 
and  credible  consultants  to  conclude  that  the  organization’s 
approach  to  its  mission  will  soon  be  obsolete,  that  the  organi¬ 
zation  should  reform,  and  that — if  it  does — it  will  be  effective 
with  fewer  resources.  In  this  case,  the  actual  worry  is  not  the 
resource  level  but  one  of  institutional  risk:  Will  the  organiza¬ 
tion  be  able  to  reform  itself? 

-  Perhaps  the  decision  was  to  reject  the  organization’s  claim 
about  “how  much  is  enough?”  based  on  judgments  about  intel¬ 
ligence  on  adversaries  and  related  independent  analysis.  In  that 
case,  the  risk  would  be  that  the  intelligence  or  analysis  was 
wrong  rather  than  the  resource  levels  per  se. 

•  When  a  quantitative  analysis  using  models  defines  risk  as  the  gap 
between  the  goal  and  the  predicted  outcome  of  a  test  case,  it  is 
“using  up”  the  term  “risk”  for  something  that  may  not  be  nearly 
as  important  as  the  possibility  of  having  the  wrong  test  case  (see 
the  first  section  of  this  appendix).  It  is  better,  in  such  case,  to 
simply  refer  to  the  gap  between  goal  and  outcome. 
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•  The  most  egregious  problem  is  failure  to  treat  “deep”  uncertain¬ 
ties  or  treating  them  poorly,  while  simultaneously  doing  in-depth 
precise  analysis  of  relatively  trivial  risks. 

•  Using  probabilistic  methods  to  estimate  the  likelihood  of  a  bad 
outcome  may  appear  elegant  and  scientific,  but  may  be  ignoring 
the  most  important  uncertainties,  assuming  independence  of  cor¬ 
related  processes,  or  relying  on  “data”  for  which  no  reasonable 
basis  exists. 


Conclusions  on  Risk 

My  own  conclusions  on  risk  management  are  (1)  it  is  preferable  to  orga¬ 
nize  around  objectives  (including  risk  reduction)  than  around  risk;  (2) 
the  many  types  of  risk  should  be  explicitly  identified  in  analysis  rather 
than  lumped  together;  (3)  the  term  “risk”  should  be  preserved  for  pos¬ 
sibilities  not  addressed  in  the  central  line  of  analysis,  as  in  “the  risk  that 
our  planning  scenarios  are  off-base”  or  “the  risk  that,  despite  promises, 
the  system  being  acquired  will  never  be  able  to  accomplish  the  goals 
assumed  in  program  “requirements”;  (4)  risk  analysis  based  on  optimiz¬ 
ing  risk  reduction  based  on  a  single  overall  metric  should  be  viewed 
with  great  skepticism;  and  (5)  probabilistic  methods  should  be  used 
only  for  narrow  purposes  and  advertised  as  such. 
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APPENDIX  D 


Illustrative  Threat-Based  Analysis 


Figure  D.l  shows  the  kind  of  build-up  chart  used  in  the  1970s  and 
1980s  for  some  requirement-setting  and  communication-related  pur¬ 
poses.  The  story  was  that  Red  (the  Warsaw  Pact)  could  build  its  forces 
faster  than  could  NATO  shortly  after  mobilization  for  war  began.  For 
a  period  of  time  in  the  first  month,  Red  could  enjoy  a  force  ratio  of 
2  or  even  3:1 — high  enough  to  make  a  quick  victory  possible.  How¬ 
ever,  if  the  United  States  could  deploy  a  number  of  divisions  quickly 
enough  (dashed  curve),  the  force  ratio  would  never  be  worse  than  1.5:1, 

Figure  D.l 

An  Illustrative  Build-Up  Chart  from  the  Cold  War 
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which  was  deemed  adequate  for  conventional  deterrence  because  it  was 
believed  that  would-be  conventional  aggressors  seek  quick  and  easy 
success.1  NATO  could  mount  a  good  initial  defense  and,  thus,  it  was 
argued  that  Red  could  not  be  confident  of  victory.2  The  program  to 
accomplish  this  involved  prepositioning  the  divisional  equipment  in 
Europe  and  airlifting  the  personnel  to  man  the  equipment  (the  famous 
POMCUS  program  initiated  in  the  1970s). 

This  was  an  illustration  of  what  purportedly  was  pure  “threat- 
based  planning.”  The  red  curve  was  based  on  intelligence  estimates; 
the  blue  curves  were  based  on  actual  NATO  capabilities  and  projected 
capabilities  with  the  POMCUS  program.  The  “problem”  was  simply 
maintaining  an  adequate  force  ratio.  The  program  solved  the  problem. 
This  type  of  analysis  was  regarded  as  clear  and  compelling. 

The  underlying  reality  of  analysis  was  more  complex.  First,  there 
was  considerable  debate  over  the  years  about  how  to  characterize  the 
strength  of  the  forces.3  The  “equivalent  division”  metric  was  imposed 
by  the  Office  of  the  Secretary  of  Defense  to  simplify  a  much  more 
complicated  situation  in  which  the  various  divisions  were  all  somewhat 
different  in  manning,  equipment,  and  readiness  level.  Second,  there 
was  controversy  about  how  quickly  Pact  forces  could  actually  build  up 
and  about  how  quickly  NATO  would  see  a  threat  and  order  its  own 
mobilization.  Third,  there  was  controversy  about  what  force  ratio  was 
sufficient — even  for  deterrence,  but  certainly  for  war-fighting.  Fourth, 
there  were  disagreements  about  whether  NATO’s  air  forces  and  com¬ 
mand  and  control  capabilities  would  tilt  the  balance  and  should  some¬ 
how  be  included  in  the  analysis.  Finally,  real  war  would  have  been 
much  more  complicated  involving  terrain,  road  networks,  generalship, 
the  role  of  airpower,  etc.  Such  matters  were  gamed  out  to  the  extent 
feasible  at  the  time,  with  results  informing  the  more  aggregate  analysis. 
Thus,  the  depiction  in  Figure  D.l  was  very  much  simplified  relative  to 
what  analysts  had  considered  and  debated  about  before  going  forward 
to  Congress.  Advocates  of  threat-based  planning  sometimes  claim  that 
such  analysis  is  simple,  clear,  and  compelling,  but  it  is  more  accurate 
to  say  that  the  final  story  was  relatively  simple,  clear,  and  compelling. 

My  own  group  in  OSD  (Program  Analysis  and  Evaluation)  used 
similar  analysis  in  1979-1981  to  assess  force  needs  for  Southwest  Asia, 
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whether  against  an  invasion  of  Kuwait  by  Saddam  Hussein  or  an  inva¬ 
sion  of  the  region  through  Iran  by  the  Soviet  Union.  In  that  instance, 
both  the  deeper  analysis  and  war-gaming  was  strongly  affected  by  the 
mountain  belts  in  Iran  and  by  whether  U.S.  forces  would  be  fight¬ 
ing  along  with,  independently  of,  or  with  Iranian  forces  in  defending 
against  the  Soviets.  Thus,  any  discussion  in  terms  of  force  ratios  and 
buildup  charts  was  as  a  much  bigger  simplification  even  than  that  for 
the  Central  Region.  The  potential  for  interdiction  was  intriguing.4  Our 
analysis  was  consciously  more  about  replacing  a  worrisome  vacuum 
with  a  deterrent  strategy  and  modicum  of  capability  than  about  pre¬ 
paring  for  likely  war.5 


Endnotes 

1  Mearsheimer,  1983. 

2  The  1.5:1  rule  of  thumb  was  dubious  and  DoD  would  have  preferred  using  a  ratio 
of  1.25:1.  The  reasoning  behind  it  was  described  in  a  Congressional  Budget  Office 
paper  (Hillier,  1980).  For  a  later  discussion  of  how  force-ratio  rules  of  thumb  change 
with  level  (theater,  corps,  tactical)  and  with  important  assumptions,  see  Davis,  1995. 

5  Until  Enthoven’s  Office  of  Systems  Analysis  looked  at  the  issue  in  the  1960s, 
assessments  treated  extremely  low-readiness  Soviet  divisions  as  though  they  were 
real  (Enthoven  and  Smith,  1971).  In  later  years,  “equivalent  armored  divisions”  was 
a  metric  that  adjusted  for  personnel  and  equipment  modernity. 

4  Levine,  1985. 

5  The  FY  1982  annual  defense  report  illustrated  the  deterrence  focus:  “By  showing 
the  Soviets  that  we  have  the  military  capability  and  the  national  will  to  respond  to 
aggression,  we  seek  to  deter  such  aggression  in  the  first  place.  The  determination 
and  ability  to  move  a  credible  American  force  rapidly  and  effectively  changes  the 
calculus  for  the  Soviets;  they  must  then  consider  the  probability  that  any  aggression 
by  them  will  meet  not  only  indigenous  forces,  but  also  those  of  the  United  States. 
Given  such  an  ability  on  our  part  to  meet  them  on  the  spot  and  our  capability  of 
shifting  the  geography  of  the  conflict,  the  Soviets  must  consider  the  possibility  that 
renewed  aggression  by  them  may  lead  to  a  much  wider  war,  escalated  both  in  inten¬ 
sity  and  geography.” 


APPENDIX  E 


A  Model  That  Could  Have  Been  "Simple" 


To  illustrate  the  trickiness  of  characterizing  a  model  as  simple  or 
complicated,  consider  Figure  E.l,  a  depiction  of  the  counterinsur¬ 
gency  problem  in  Afghanistan  that  appeared  in  the  background  of  a 
New  York  Times  article  entitled  “We  Have  Met  the  Enemy  and  He 
Is  PowerPoint.”1  It  is  reported  that  General  Stanley  McChrystal 
dryly  remarked,  “When  we  understand  that  slide,  we’ll  have  won  the 
war.”  The  article  is  worth  reading,  and  many  readers  will  appreciate 
the  grumps  about  death  by  Power  Point.  However,  what  is  perhaps 
interesting  is  that  what  was  being  shown  was  a  visual  depiction  of  a 
model  that,  in  some  respects,  is  fairly  simple — even  if  the  impression 
is  precisely  the  opposite  and  even  if,  had  there  been  an  objective  test 
of  comprehension,  the  result  for  most  audiences  would  have  been  a 
total  lack  of  comprehension — beyond,  perhaps,  acknowledging  that 
everything  is  related  to  everything  in  something  as  complicated  as 
counterinsurgency. 

Just  as  campaign  models  can  be  considered  complex  and  opaque, 
or  rather  simplified  and  revealing,  depending  on  one’s  expertise  and 
how  the  model  is  presented,  so  also  the  model  underlying  the  graphic 
can  be  seen  as  a  remarkable  distillation  of  a  great  deal  of  knowledge 
that  can  be  quite  understandable — given  some  investment  of  time.  The 
underlying  model  was  developed  in  the  System  Dynamics  modeling 
platform  by  the  PA  Consulting  Group  and  the  Joint  Staff5 s  J-8.2  It  was 
based  not  on  complex  mathematics,  but  on  the  Petraeus-Mattis  field 
manual.3  The  effort  to  develop  the  model  had  been  highly  informative, 
as  was  the  briefing,  which  was  given  numerous  times  to  analysts  and 
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Figure  E.1 

The  Famous  "HairbaN"  Graphic 
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operators.  The  operators  reportedly  found  the  diagram  very  useful,  but 
had  little  or  no  interest  in  the  running  model — not  just  because  it  was 
below  their  level  of  resolution,  but  because  they  knew  that  it  was  the 
ability  to  see  the  overall  system  and  how  interactions  occurred  that  was 
insightful. 

Without  elaboration,  let  me  simply  assert  that  the  model’s  opaque¬ 
ness  was  largely  an  artifact  of  having  been  presented  “flat”  rather  than 
in  layers.  Had  the  model  been  designed  with  multiresolution  meth¬ 
ods  and  presented  accordingly,  the  top-level  diagram  would  been  far 
simpler,  and  the  primary  points  to  be  made  could  have  been  made  in 
digestible  chunks — rather  like  the  zooming  discussed  in  Figure  6.2  for 
portfolio  analysis  or  in  the  multiresolution  modeling  of  public  support 
for  terrorism  mentioned  in  the  text  as  one  example  of  simple  modeling 
(Davis  and  O’Mahony,  2013). 


Endnotes 

1  Bumiller,  2010. 

2  The  authors  used  the  VENSIM  and  STELLA/iTHINK  programming  languages 
for  diagraming  and  coding,  respectively. 


3  Petraeus  and  Amos,  2006. 


APPENDIX  F 


An  Example  of  Simplified  Modeling:  The  Halt 
Problem 


This  appendix  illustrates  how  it  is  sometimes  possible  to  base  a  useful 
parametric  model  on  a  simple  physical  picture  and  freshman-level 
mathematics.  It  is  perhaps  most  common  to  think  of  simple  models  as 
a  place  to  begin,  to  sketch  out  ideas  before  doing  “more  serious”  model¬ 
ing.  The  following  example,  however,  describes  instead  the  instance  in 
which  one  starts  with  a  complicated  computer  model  but  finds  it  diffi¬ 
cult  to  work  with  or  explain,  and  of  little  help  in  identifying  shortcom¬ 
ings  of  the  analysis  that  should  be  addressed.  In  that  case,  reverting  to 
simple  modeling  can  bring  clarification  and  insight. 


Background 

In  the  mid  to  late  1990s,  a  number  of  organizations  used  simulation 
modeling  to  examine  whether  precision  fires  could  improve  the  ability 
of  a  defending  force  to  halt  an  invading  army.  Much  such  analysis  was 
complicated  because  it  used  large  campaign  models  with  considerable 
detail  on  ground  forces,  air  forces,  deployment  processes,  terrain,  and 
so  on.  By  running  a  case  with  and  without  the  assumption  of  some 
precision-fires  options,  outcomes  could  be  compared,  but  it  was  diffi¬ 
cult  to  understand  the  essence  of  what  was  happening  unless  one  were 
deeply  expert  with  the  model,  and,  even  then,  it  was  difficult  to  answer 
some  questions  because  outcomes  depended  on  so  many  variables.1 
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A  Simple  Underlying  Model 

Upon  stepping  back  from  the  campaign  models,  it  became  evident 
that  outcomes  were  being  driven  by  a  simple  process  that  could  be 
isolated  from  the  rest  of  the  campaign  model.  The  assumed  process 
was  that  the  armored  vehicles  of  mechanized  divisions  moved  down 
a  road  network  and  were  being  attacked  by  precision  fires  from  fixed- 
wing  or  rotary-wing  aircraft,  or  from  postulated  long-range  precision 
artillery  The  “shooters”  were  credited  with  the  ability  to  kill  a  certain 
number  of  vehicles  each  sortie.  A  mechanized  unit,  such  as  a  division, 
was  assumed  to  become  useless  (and  essentially  stop)  after  suffering  a 
sufficiently  high  level  of  attrition,  such  as  50  percent.  The  underlying 
picture,  then,  was  just  a  race  between  the  moving  vehicles  and  the  attri¬ 
tion  of  those  vehicles. 

In  the  tradition  of  simplified  modeling,  it  was  possible  to  assume 
that  the  invading  army  moved  at  a  constant  speed,  and  that  interdic¬ 
tion  could  be  represented  roughly  by  an  aggregate  number  of  shooters 
with  an  average  killing  rate  per  sortie  and  an  average  sortie  rate  per 
day.  The  number  of  shooters  on  a  given  day  could  be  represented  as 
the  number  of  shooters  available  at  the  outset  plus  those  being  added 
by  deployment  at  an  average  deployment  rate.  A  successful  halt  cam¬ 
paign  would  kill  a  critical  number  of  armored  fighting  vehicles  before 
the  invading  army  had  penetrated  some  unacceptable  distance  into  the 
target  country.2  This  glossed  over  many  subtleties  and  complications, 
but  allowed  “seeing  the  whole”  in  a  mathematical  sense.  Further,  the 
simple  problem  was  solvable  in  closed  form,  as  shown  below,  using  a 
trivial  integral  equation  that  one  might  understand  from  freshman  cal¬ 
culus  and  the  famous  quadratic  formula  of  eighth-grade  algebra. 

Th  , 

FN  =  f  KS{A0  +  Rsjsds  =  KSA0 (Th  -  Ts )  +  - KSR(Th  -  Ts )2 

J  2 

TS 

-KSA0  ±  ^/(KSA0)2  +  2KSA0FN 


•••Th=Ts  + 
D  =  VTh 


KSR 
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where 

F  is  the  fraction  of  the  attacking  force  that  must  be  killed  to  bring 
about  a  halt, 

N  is  the  number  of  armored  vehicles  in  the  attack, 

K  is  the  number  of  vehicles  killed  per  sortie, 

S  is  the  number  of  sorties  per  day  (for  aircraft), 

R  is  the  rate  at  which  shooters  deploy  into  the  theater, 

Aq  is  the  initial  number  of  shooters, 

Th  is  the  time  at  which  the  halt  occurs, 

Ts  is  the  time  required  to  suppress  air  defenses  so  that  interdiction 
can  proceed, 

D,  the  distance  of  penetration  before  halt  occurs,  is  the  average  rate 
of  movement  times  the  halt  time. 

The  first  equation  states  that  the  required  number  of  vehicles  will 
have  been  killed  after  interdiction  has  proceeded  from  time  Ts  to  time 


Discovering  Assumptions  to  Be  Varied 

To  be  sure,  the  simple  model  in  the  equation  above  was  making  “errors” 
aggregating  even  more  than  the  campaign  model  itself — by  referring 
to  shooters  rather  than  a  number  of  different  shooter  types  with  some¬ 
what  different  killing  effectiveness,  sortie  rate  (or  shot  rate),  and  deploy¬ 
ment  rate.  The  campaign  model  would  distinguish  among,  say,  F-15s, 
F-15s,  F-18s,  and  long-range  artillery  with  advanced  munitions.  FIow- 
ever,  these  numerical  “errors”  (actually,  just  approximations)  were  triv¬ 
ial  in  comparison  with  uncertainties  about  the  basic  picture  embodied 
in  the  initial  campaign  model  as  well  as  the  simplified  model.  Even 
the  full  campaign  model  assumed,  initially,  that  the  invading  army 
kept  moving  at  constant  speed  despite  attrition  until  “destroyed,”  that 
“destruction”  occurred  when  a  certain  fraction,  F,  of  the  vehicles  were 
killed,  that  aircraft  would  be  able  to  operate  immediately  rather  than 
waiting  until  after  air  defenses  were  suppressed,  that  the  kills  per  sortie 
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or  shot  metric  was  unaffected  by  the  invader’s  maneuver  strategy  (e.g., 
vehicle  spacing,  the  number  of  roads  used),  and  that  bases  were  avail¬ 
able  to  accommodate  deploying  shooters.  Each  of  these  inputs  was 
uncertain  by  at  least  a  factor  of  two! 

If  these  underlying  assumptions  were  pointed  out,  the  campaign 
model  could  readily  be  adjusted  to  permit  variations.  However,  in  some 
cases,  it  was  the  simple  modeling  that  laid  bare  the  assumptions.  Further, 
it  was  often  easier  and  faster  for  an  analyst  to  adjust  the  simple  model 
to  accommodate  different  assumptions  than  to  have  a  professional  pro¬ 
grammer  make  the  lower-level  coding  changes  in  the  campaign  model. 
And,  perhaps  most  important,  when  questioning  the  assumptions,  it 
was  easier  to  think  in  terms  of  the  simple  model  when  conceiving  alter¬ 
native  strategies  for  both  the  invader  and  the  interdictor.  Literal  back- 
of-the-envelope  thinking  (or  white-board  thinking)  suggested  that  the 
interdictor  might  focus  its  efforts  on  the  leading  edge  of  the  invad¬ 
ing  force,  in  which  case  attrition  would  not  only  destroy  vehicles  but 
slow  the  rate  of  the  column’s  advance.3  On  the  other  hand,  the  invader 
could  increase  the  spacing  between  vehicles  and  use  more  roads,  both 
of  which  would  be  expected  to  reduce  the  effectiveness  of  interdictors. 
At  a  higher  level,  suppression  of  air  defense  might  take  time  (hours  or 
a  number  of  days)  and  airports  might  not  have  the  capacity  to  accom¬ 
modate  new  aircraft  or  might  themselves  be  under  attack,  delaying  the 
time  at  which  the  aircraft  could  be  accommodated.  Also,  the  number 
of  shooters  available  initially  and  the  deployment  rates  subsequently 
should  depend  sensitively  on  the  nature  of  warning.  Finally,  it  might 
be  important  to  consider  the  potential  value  of  a  blocking  force  on 
the  ground,  even  a  relatively  small  one.  All  of  these  issues  could  be 
represented  easily  with  a  simple  spreadsheet  model  or  its  equivalent  in 
another  high-level  language. 

One  consequence  of  this  period  of  work  at  RAND  was  that 
RAND’s  Project  AIR  FORCE  spun  off  from  the  Joint  Integrated  Con¬ 
tingency  Model  campaign  model  a  spreadsheet  model  called  START 
that  focused  on  the  interdiction  problem  while  representing  air  forces 
in  adequate  detail  for  RAND’s  analysis.  In  effect,  this  amounted  to 
“discovering”  a  module  of  the  campaign  model,  splitting  it  off,  and 
reprogramming  for  convenience.  START  has  been  used  frequently  in 
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the  intervening  years.  For  other  analyses,  much  more  detail  is  needed, 
and  a  campaign  model  such  as  the  Air  Force’s  STORM  model  is  often 
used.4 


Parametric  Analysis 

The  next  great  strength  of  the  simplified  modeling  was  exploratory 
analysis,  i.e.,  analysis  in  which  all  of  the  important  input  assumptions 
are  varied  simultaneously  so  as  to  generate  parametric  plots  such  as 
shown  in  Figure  F.l,  an  example  showing  how  outcomes  (halt  distance) 
varies  as  a  function  of  five  variables.  This  is  hardly  ordinary  sensitivity 
analysis,  in  which  one  assumes  a  baseline  case  and  then  does  excursions 
around  it,  often  varying  only  one  assumption  one  at  a  time.  It  does 
not  assume  any  particular  base  case,  and  it  shows  important  interac¬ 
tions  among  the  variables  of  a  problem.  For  example,  the  lack  of  stra¬ 
tegic  warning,  or  even  the  lack  of  tactical  warning,  can  be  devastating 
because  of  effects  on  all  aspects  of  the  interdiction  effort.  Such  work 
can  be  used  for  trade-space  analysis  and  other  purposes. 


Takeaways 

The  intent  of  this  discussion  has  been  to  illustrate  how  simple  mod¬ 
eling  can  (1)  clarify  how  the  underlying  phenomenon  is  being  con¬ 
ceived  within  a  more  complex  treatment,  (2)  be  a  convenient  vehicle 
for  the  analyst  to  use  when  for  relaxing  or  varying  assumptions  about 
phenomena  and  factor  values,  (3)  be  the  basis  for  broad  parametric 
analysis,  and  (4)  be  a  way  in  which  to  sharpen  requirements  for  how 
the  more  complex  model  should  be  enriched.  These  virtues  of  simple 
models  have  been  exhibited  in  countless  studies  over  the  decades,  but 
simple  models  are  underused  today  because  so  much  attention  and 
education  focuses  on  more  complicated  computer  models. 
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Figure  F.1 

Results  of  Exploratory  Analysis  with  Five  Variables 
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NOTE:  The  charts  show  how  far  the  invader  moves  before  being  stopped,  the  halt 
distance  (vertical  axis),  as  a  function  of  the  number  of  shooters  available  on  D-Day 
(horizontal  axis),  the  kills  per  shooter  per  day  (left  and  right  sides  of  the  each  chart),  the 
time  required  for  suppression  of  air  defenses  before  the  shooters  can  operate  (the  white 
versus  black  bars),  the  number  of  vehicles  to  be  killed  before  stopping  the  invading  force 
(^,  which  is  different  for  the  first  and  second  chart,  reading  downward,  and  for  the  third 
and  fourth  chart),  and  the  average  speed  of  the  invading  columns  (V,  which  is  half  as 
much  for  the  bottom  two  charts  as  for  the  tow  two  charts).  If  a  halt  distance  less  than 
1 00  km  were  sought,  then  it  is  obvious  from  a  quick  perusal  that  TSEAD,  the  time  to 
suppress  air  defenses,  must  be  small,  and  that  either  the  average  movement  rate  must 
be  small  or  the  number  of  D-Day  shooters  and  the  kills  per  shooter  day  must  be  large. 
AFV  refers  to  armored  fighting  vehicles. 
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Endnotes 

1  My  own  experience  is  mostly  with  RAND’s  Joint  Integrated  Contingency  Model 
(Jones  and  Fox,  1999),  largely  built  by  colleague  Carl  Jones,  with  mechanisms  per¬ 
mitting  exploratory  analysis  (Fox,  2003). 

2  See  Davis  and  Carrillo,  1997,  for  early  exploratory  analysis.  An  influential  study 
the  next  year  for  the  Air  Force  used  a  simple  simulation  model  (Ochmanek  et 
al.,  1998).  Much  richer  analysis  was  published  later  and  included  extensive  explor¬ 
atory  analysis  across  major  strategic-level  uncertainties  bearing  on  the  access  issue 
(McEver,  Davis,  and  Bigelow,  2000;  Davis,  McEver,  and  Wilson,  2002). 

1  The  leading-edge  interdiction  strategy  was  suggested  by  my  late  and  legendary 
colleague  Glenn  Kent  (Lieutenant  General,  Ret.,  USAF).  A  comparison  of  results 
with  and  without  the  benefits  of  this  strategy,  which  might  or  might  not  have  been 
feasible  depending  on  air  defenses  and  counters,  is  included  in  a  2002  study  (Davis, 
McEver,  and  Wilson,  2002). 

4  STORM  is  a  campaign-level  analysis  simulation  developed  by  Group  W  with 
offices  in  Fairfax,  Virginia,  and  Triangle,  Virginia. 
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Defense  analysis  can  do  a  better  job  supporting  policymakers  dealing  with 
multiple  objectives  and  deep  uncertainties.  This  will  involve  seeing  through  the 
fog  with  simple  analysis  and  undergirding  results  with  depth  as  necessary.  It  will 
emphasize  balancing  across  objectives  and  hedging  against  both  uncertainty  and 
disagreement  among  policymakers.  Modern  methods  for  doing  so  are  available 
but  they  require  displacing  some  familiar  processes  and  demanding  more  from 
analysis.  Once  decisions  are  made,  analysis  should  help  policymakers  explain, 
convince,  and  shape  implementation  guidance  with  sharpened  requirements, 
forcing  functions,  and  metrics  for  monitoring,  feedback,  and  adaptation. 
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